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EXECUTIVE SUMMARY 



When Adam walked with Hiv iti the Garden of fuk'n, he 
iras overheard to say (presumably hy the angel just arrived 
with the flmning sivord ), "You must understand, my dear, 
that we are living through a period oj transition" (Gm- 
19S1. p. 213 V 

Tliis \x)lume is also about transitions and issues ..ni'okiing 
during the impiementafion of performance indicators. Begin 
ning in the 1980s, the era concerned primarily with grov^th 
in enrollments and access was largely over, while another 
waited further definition and recognition in such emerging 
issLies as public accountabiiirs', quality, productivity, and 
undergraduate educatit)n. 

The 1980s was also distinguished by the growth of the 
nun'ement U)ward assessment and accountability. While 
higher education in the United States was affected by several 
phenomena during this decade, surely none created more 
funckmientai change thai', the movement tt)ward assessment. 
A 1990 study by the Education Commission of the States, ior 
example, rcwaled that 40 .states actively promoted a.s,se,ssment- 
(Eweli, I'innc}', and I^nth 1990). Alt)ng with this movement 
was a rising intere.st in the quality of undergraduate education, 
and a litany of .studios published in the 1980s lamented the 
poor condition of undergraduate education, pointing to 
inadequacies that needed to be corrected (.see. e.g., As.st) 
elation of Axiierican 198''; Bennett 1984; Boyer 198^; U'vine 
and Boyer 1981: Sykes 1988). By 1986, all 50 .states and the 
Di.strict of Ct)lumbia had devck)ped initiatives to improve 
undergraduate education (Boyer and McGuinness 1986). 

Accompanying this mo\'ement was a subtle shift from 
grtn\th in funding, principally through formula funding, 
toward funding "outcomes," "results." and "performance." 
Tliis f(K"us on iierformance, using funding incentives as mo 
tivators, helped encourage |X)li(.y makers and the academic 
community to explore the use of a .system of indicatt)rs to 
r;iise warning signs about the efficiency and effectivene.ss of 
higher educatit)n. 

Iliest- ck)me.stic efforts paralleled developments in higher 
education in a number of countries, particularly in Europe 
and Au.siralia. Since the late 19''0s. the concepts (jf perfor- 
mance indicators and quality a.s.se.ssment ha\e clearly become 
international i.ssues (Dtx-hy, Segers, and Wijnen 1990a, 1990b; 
Kells 1993). Indeed, they are becoming an integral part oi 
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an emerging intemutional method on bow to manage liigher 
education, with indicators ser\'ing as signals or guides for 
making national or international comparisons in educational 
quality, effectiveness, and efficiency- (National Center for Edu- 
cation 1994a; Sizer, Spec, and Bormans 1992). Further, the 
main advantage of such performance indicator systems is their 
usefulness as points of reference for comparing quality or 
performance against peers over time, or achie\ement against 
a desired objective. 

Tfie 1990s emerged as part of another era awaiting further 
definition. First, the development of performance indicators 
in the 1990s differs from that in the 198Gs. Polity makers arc 
generally less inclined toward the voluntary institutional 
improvement of the 198Gs and more focused on a system 
of mandated public accountability. And by 1994 some 18 
states had developed indicator systems, most of them in the 
first three years of the decade (Ewell 1994a). A heightened 
tempo in the use of performance indicators, accompanied 
by a tendenty to copy other states' sAstems, resulted in a com 
mon core of state indicators to address common problems. 
Concomitant with this movement was greater centralization 
of authority, with the intent of bringing about more public 
accountability and better management— which will likely 
underlie much future funding of higher education in the 
United States. 

Tlie air is full of questions. Will the federal government 
assume greater centralized control of higher education 
through such areas as accreditatic^n and financial aid and by 
I'sing a set of national goals and performance standards? Will 
international education continue to be reformed through the 
mechanisms of performance indicators and incentive funding? 
How should such mechanism;; be be.st u.sed to motivate and 
bring about desired reforms on campus and at state, regional, 
or national levels? ^M^ile scholars and legislators debate these 
questions, the public's investment in and concern about qual- 
ity and perfc^rmance in higher education continue unabated, 
and institutional resistance to fundamental reform remains 
ingrained. It remains unclear whether performance indicators 
and incenti\-e funding will result in any widespread, lasting 
innovations or the concept will pa.ss (.juickly through higher 
education in this country, leaving only a modest residue. 

Perhaps, however, a hint about any la.sting contribution 
and the future role for performance indicators can be found 
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in Europe, wliere early pioneering efforts on quality assess 
nicnt are maturing. Nationally, the role of performance indi 
cators is declining (Green 1994; Jongbloed and Westerheijden 
1994), and growing doubts about the ability to '"measure the 
unmeasurable," particularly about the \ alidity of such mea- 
sures to evaluate and be used to reward cjuality, have led to 
retrenchment in such countries as the Netherlands and the 
United Kingdom. At the same time, national and in.stitutioiial 
experiments with such a.ssessment techniques as peer reviews 
and qualit>' audits are gaining prominence, relegating per- 
formance indicators to the role of supporting tools in such 
efforts (Green 1994; Jongbloed and Westerheijden 199-h). 

Tliis emerging approach offers the collective faculty a more 
palatable, more dwamic vision of academic quality, o.stensibl\ 
more worthy of their commitment and pursuit than any exter- 
nally imposed system of performance indicators (Burke 1994). 
Faculty resolutely insi.st they know academic quality when 
they see it and .should retain the primary- responsibility for 
assessing and rewarding it. But such autonomy is always pur 
chased by providing measures of accountability for results 
and resources to the public and to policy makers. It remains 
lo be seen wiiether faculty will a,ssume the collecti\'e mantle 
of responsibility antl professional obligation to develop pro 
ce.s.ses that develop a sense of common purpose and shared 
accountability with the various publics. If this pattern gains 
prominence, performance indicators will likely be relegated 
to a minor role as a supporting tcx)l; if the academy does not 
respond, the public appetite for results v^ill expand and ciys- 
talli/.e around the use of external performance indicators to 
measure desired results. And the jur\- is still out on the results 
desired. 
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FOREWORD 



The increasinj^ demand for quantifiable evidence that liigher 
education is worth tiie cost and the time involved is a result 
of three major economic and scx-iai changes of the last three 
decades. First, a higher education has become considerably 
more import in securing and keeping a good paying job than 
ii \v;is 30 years ag<.). For scK'ieiy as a whole, the quality of 
higher education institutions is cmcial to the nation's ability 
to compete economically with the other countries of the 
world. Second, consumers have become more sophisticated. 
The? more college graduates and the more .students with par- 
ents who are college graduates, the less willing are consumers 
to accept the effectiveness of their education on faith, espe 
cially when they believe their personal expectations for an 
educatit)n are noi being met. .Students arc no longer willing 
to accept that the faculty's professional expertise should allow 
them to make decisions that affect their lives without .some 
evidence of the \ alidity and reliability of those decisions. 
Third, the cost of higher education is greater than it was 30 
years ago. Higher education is taking an increasingly large 
pt)rtion of disposable or di.scretionar\- income of both indi- 
\-iduals and public funding agencies. As decisions about 
spending become harder and more painful to make, people 
naturally want information to help them decide where they 
should put their money. Tlie consequence is greater demand 
for quantitative data about how well higher education is per 
forming. Hut v,hat should be measured, and lu)w will this 
measurement be used? 

Tliree t>pes of performance indicators can be used to judge 
the cjuality of any prcKcss. Tlie t\pe mo.st often used is th.e 
measurement of results - for higher educatitm, for example, 
the percetitage of .students graduating, the number of degrees 
awarded, the percentage of .students going on to graduate 
.sc1k!o1s, or the number of publications by faculty. Tliese mea 
sures do not indicate a sense of quality, however, onK' tjuan 
tit\'. hicreasingK-, the expectation is now to measure the qual 
ity or degree of the outct)nie-- "measuring the value added" - 
exemplified by h.ow many .students went to the best graduate 
.schools, their average .starting salar\' compared to comparable 
.schools, or the number of faculty publicatit)ns cited in the 
publications of other .scholars. 

Tlie second measure of performance is the tjualit)- of 
inputs: SAT or ACI" scores t)f entering .students, their rank in 
cla,ss or recognition as a merit .scholar, the rank of faculty 
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members" graduate degree programs, whether faculty brought 
with them research grants, or the i j iber of referoed pub 
lications. 'llic thc()r\' behind indicators of this t^pc is that if 
the quality of input is high, then results will be high. 

The third set of measurements that has become increasingly 
important as performance indicators is measurement of or 
at critical prcK'css [XMnts. Tliis new emphasis is a result of con- 
cern with the c)ther two t\pes of measurement. Wlien only 
outct)mes or results arc measured, then it is too late to do 
an\,thing but accept or reject the situation. For many, the real- 
ization that 50 percent of starting students never achieve their 
education goals is no longer acceptable. Research has shown 
that if a prwess is flawed, highly satisfiicror^' results will not 
Ix' ensured even if the process .starts with quality .students 
and faculty. Con.sequently, the prcKcss of education is receiv- 
ing increasing scaitiny through indicators such as how often 
.students u.se the library' ^ach week, participate in .study 
groups, hold positions of leadership, write papers, or consult 
with their academic adviser. For faculty, prcKe.ss indicators 
could be the number of hours they u.sc to prepare for cla.ss, 
their use of new technology in teaching, or how they validate 
their ;Ls,se,ssment of students. 

The demand for and use of performance indicators are the 
foci of McMui nif^ ( 'p: The Promises and Pitfalls of Per/or 
maiice Indicators in Higher Education. Gerald Gaither, direc 
tor of in.stitutiona! research at the Prairie View campus of the 
Texas A&M LIniversity sy.stem, Brian R Nedwek, a,s,S(K"iate pro- 
vo.st and professor of public policy studies at Saint L(.)uis I'ni 
versity, and John E. Neal, a.s.s(Kiatc vice president at Webster 
I'niversity in Saint Ijouis. have s">aithesi/.ed the mcxst important 
literature concerning the use of various performance indi- 
cators and their po.ssilile shortcomings. Tliev look at the expe 
riences of higher educatic/n in the United States and then 
review the development and u.se of perfonriance indicators 
in Great Britain, Canada, Australia, the Netherlands, Finland, 
Sweden, and [Denmark. The authors' conclusions provide a 
direction that could hel[") higher education in.stitutions in this 
countiy develop more effective ways to measure the quality 
of their efTort.s. 

The trend is clear and irre\X"rsible: Higher education in.sti- 
tutions mu.st .start dev eloping better ways to judge more ade 
ciuately how well they arc doing. Higher education has been 
reluctant to develop performance indicators becau.se, it is 



believed, the mission of higher education is too diverse to 
measure and short-term measurement might not he adequate 
to measure long-term success. If the members of the 
academy— faculty, academic leaders, students— do not par- 
ticipate in the process of developing and improving the use 
of performance indicators, however, external organizations 
will force st)me form of indicators on them. Tliis report pro- 
vides a broad perspective that will be useful in esiabli.shing 
an understanding of the issues surrounding the use of per- 
formance indicators. 

Jonathan D. Fife 

Series Editor, Profe.ssor of Higher Education Atlmini.stration, 
and Director, ERIC Clearinghouse on I ligher Education 
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THE POLICY CONNECTION: Ideas and Actions 



All inf(isk»i of armies can he rcxisWd, hut not an idea 
whose time h^is come. 

— Victor Hugo, Histoire d'ttn Crime 



Some Background Issues 

State- level quality incentives and performance indicators are 
receiving increased attention nationwide among higher edu- 
cation polity makers and planners. Consider: 



• Under the rubric of "the accountability imperative," sev 
eral states ha\'e now passed laws that require institutions 
to demonstrate effectiveness, efficiency, .sound fiscal stew 
ardship, and proven perfonnance (Ewell 1990, 1994a; 
Lively 1992). 

• A 1990 .study by the Education Commission of the States 
reveals that 40 .stiites actively promoted asses.sment (Ewell, 
Finney, and I^nth 1990). 

• A later report by the AjTierican Council on Education notes 
that 97 percent of all in.stitutions claim to engage in assess- 
ment (Marchese 1994, p. 4). 

• As of 1992, 19 states required public colleges and uni- 
versities to set up programs to assess what students leam 
in college ( Chronicle 1992). 

• Eleven of the IS .states in the Southern Regional Education 
Board (SREB) have pas.sed legislation in the pa.st decade 
related to a.sse.ssment, quality, and/ or performance report- 
ing (Uvely 1992). 



'Hie i.ssues of quality and performance indicators have cap 
tured the attention of those in higher education, revealing 
a certain tension among t:ampu,ses, .st;ites, and the public. 
Prompting much interest in these topics are, among other 
things, reduced pulilic confidence in education, shrinking 
.state budgets, taxpayers' complaints about rising co.sts and 
ULxes, and concern at both the state and national levels about 
the kxss of economic competitiveness, partially ;is a result of 
the perceived erosion in educational quality. (Although public 
confidence in primarily all large organizations in the United 
.States has eroded in an absolute smsc since the beginning 
of the 1980s, organizations like organized labor, organized 
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religion, and congress have generally suffered a much grea.er 
^■c'/ci^/rc' erosion of publi': confidence than has education.)* 

Wliile a large gap exists on many campuses betv^'cen these 
legal mandates and the reality of assessment, the move toward 
assessment has helped create a notion of excellence based 
on goal oriented performance, sometimes rewarded with 
funding. Tennessee's classic goal-oriented performance fund- 
ing model, v.'hich was legislated in 198-4, is still a prototype 
for other states. Indeed, the SREB helps to prepare model leg- 
islation to allow states to develop "good practices" with 
respect to accountability and educational quality, particularly 
at the undergraduate level. As early as 1987, the state of Texas 
moved toward consideration of incentive funding, and later 
performance funcing, as part of this "gocxi practices" effort 
(Ashworth 1994; Bateman and Elliott 1994; Gaither 1993). 
Arltansas is developing a system of productivity goals, Missotiri 
is developing performance funding under the aegis of "fund- 
ing for results," and .South Carolina and Virginia use "report 
cards" on effectiveness, ostensibly logical first steps along tiie 
road to incentive or performance funding. Otitside the SREB, 
Colorado in 1993 adopted a form of incentive funding to 
rew;ird institLitions for exemplary performance that matches 
.state goals (Lively 199-+). As of 1994, some 18 states had a sys 
tern of .state based higher education performance indicators 
in place, most of which had been de\'eloped wif hin the pre- 
\ ious three years (Ewell 1994a, p. 147), and it seems likely 
that this approach will be used increasingly across the country 
(.Ashworth 1994, p. 11). 

Tliese related concepts of assessment, performance indi- 
cators, and performance fimding have the same intended goal: 
to retiuire institutions of higher education to demonstrate 
accountabilit)- and achievement of their missions and goals. 
They are often also designed, howe\'er, to influence univer- 
sities' and colleges' priorities and their achievement of their 
mission as teachers of undergraduates. Ihe external attention 
fovused on these concepts has fueled an internal campus 
dri\'e for better academic performance, p;irticiilarly in the 
delivers- of undergraduate teaching (F3ogue and Saunders 

1992; Forum 1990). In response to a litany of .'•■tudies pub 

'Tom Smith, diri'i'tni. tlcncrui Si.)t.-i.il SLin.'L'y, Naiimial Opiniim Rcscarcli Ccn 
HT. I 'iiiMTsity of Chiiagi) (^DrrcspoiukMUv .May 2H, !W3. aiul icIephoiK.- am 
wrsalmii Mari-h 2.V 



lished in the 1980s lamenting the ptxir condition of under- 
graduate education and pointing to inadequacies needing 
correction (see, e.g., Association of,\merican 1985; Bennett 
!98-4; Boyer 1987; Levine and Boyer 1981, Syke;-, 1988), a 
report on undergraduate education by the Education Com- 
mission of the States (1990) calls for each institution to de- 
velop "its own indicators of effectiveness" that would reflect 
the "institution's distinctive undergraduate mission" ajid pro- 
vides a list of suggested indicators to afford "some compara- 
bility across institutions of similar missions" (see also Boyer 
et al'. 1986 and Evangelauf 1986). 

Public concern, particularly about the quaiit)' of instruction, 
reached go\'ernors and legislators, and by 1986 ali 'lO states 
and the District of Columbia had developed initiati^■es to 
impro\'e undergraduate education (Boyer and McGuinne.ss 
1986). The public's interest in linking funding with .specific 
results also lielped prompt legislators to act. "Much of the 
impetus behind (such) assessment is the desire by state lead 
ers to hold colleges and universities accountable for achieving 
their educational puq^wes and for expending their funds" 
(Gaff 1991, p. 58). Karlier management .systems and their 
buik-in llinding motivators rewarded grcwth and .spending 
o\'er effectiveness and efficiencA-. for when government agen- 
cies, including public higher education, receive their appro- 
priations on input-t\pe Rinding formulas, "they have little -ea 
son to strive for better performance" (Osborne and Gaebler 
1993_ p. 140). Hie resulting shift, alnuxst imperceptible in 
some quarters, was toward "results-oriented government"- - 
that is, ftinding outcomes, not inputs— and words like "per- 
formance" and "results" iiegan to enter the lexicon of govern- 
ment (p. 349) (see. e.g.. Barrett :ind Greene 1992; McKenna 
1991, 1993;Shaqi 1993)- 

This fcK-us on performance and incentives helped enccuir- 
age policy makers and the academic community to explore 
the use of indicators to measure performance- or at lea.st to 
rai.se warning signs abcuit the efficiency and effectiveness of 
education in ser\-ing the needs of its \'arious cor.siituencies. 
In particular, increasing emphasi.'^ has been placed over the 
pa.st decade on the formal a,s.sessment of in.stiiutional ix-rfor 
mance and the u.se of funding, usually in iimiied amounts, 
as an incentive and reward for high performance and achieve 
ment of state goals (Brinkman 1982; Green 19H-<; Howell 
198':'; Hvatt and .Santiago 1984; Hyde 1983; McMahon 19H6). 
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Institutional assessment without a link to funding had only 
limited success and was not as fiir reaching as politY makers 
desired or anticipated, !n the final analysis, incentive funding 
programs in the 1980s v.'ere accepted in only modes; doses 
as components of strategic change and quality reform (Ho! 
land and BerdaJil 1990). Growing fiscal problems in se\'eral 
suites in the late 1980s added to the demise of such programs; 
in New Jersey, for example, revenue shortfalls had eroded 
incentive Rinding by 1990 (Hollander 1991). 

Tliese domestic effons parallel general developments in 
higher education in a number of countries, particularly in 
Europe (Great Britain and the Netherlands) and Australia 
(Huftier and Rau 1987; Keil;: 1992b, 5993; Sizer. Spee, and 
Bormans 1992), and since the late i970s, the concepts of per 
forinance indicatofs and quality a.s.sessmeni ha\'e clcarlv 
become international issues (I)ocliv, Segers, and \X ijnen 
1990a; Kells 1990, 1992h, 1993). Tney are becoming an inte- 
gral part of an emerging international method on how to bet 
ter manage and assess higher education, and they ser\'e as 
signals or guides for making national or international com- 
parisons in educational quality, effectiveness, aiv.t efficiency 
(Burke 1993a, p. 28; National Center for Education 1994a; 
Si/.er, Spee, and Bormans 1992). in Europe and Australia, the 
use of indicators generally remains at the -.lational level, while 
in the United States, performance indicators have remained 
largely the province of the stales. Tliat pattern could change 
in the 1990s, however, if federal edicts are (ieemed nece,ssar>- 
to supplement lagging .state and institutional efforts to provide 
a concept of quality based on performance and results ( Edger- 
ton 1991). 

Tlie concepts of assessment produced in Australia, the 
Netherlands, and the United ICingdom were primarily the 
product of government ideology in the context of quality 
a.ssessment; such indicators have tended to reflect the coun 
try's and the period's ideologies, and they were developed 
in respon.se to educational i.ssues of the day (Darling 
Hammond 1992). Such gcn'ernment efforts to use indicators 
as policy le\'ers or methods of control are ill-advised, and to 
be effective, they mu.st first reflect educatifjnal activities and 
institutional proce.s.ses if they are to ,sen.'e a larger, more con 
.stnictive policy role (Dading Hammond 1992 ). Otheis em 
phasized the ii.se of such indicators in measuring "goal con 
gruence" between the in.stitutions and government, suggesting 




that indicators c-ould be best used as "indications of perfor- 
mance" and not precise statistical measures (Sizer, Spee, and 
Bormans 1992). The emphasis in this volume is on the results 
of such indicators as "signals," not absolute statistical meas- 
urements. Thus, the term "performance indicators" seems 
preferable to "performance measures," because the latter 
invests the concept with an impossible precision. State and 
institutional asse.ssmcnt goals in the United States might not 
he as fai- apart as often assumed (Richardson 1994a); thus, 
the initial tensions brought about by the introduction of qual- 
ity indicators could be resolved with time. 

Tlie .specific reasons for adoption of these concepts varied 
from countr\- to countiy and from .state to .state. In Australia, 
for example, performjuice indicators were developed to 
cncxuirage education to better respond to government prior- 
ities (DcK-hy, Segers, and Wijnen 1990a; Kells 1992b). The 
government embraced the concept with such enthusiasm that 
the Au.stralian Vice Chancellors Committee and the Asswiation 
of College Directors and Principals i.ssued a word of caution: 

Something ivsonblhig a. . . cult seems to have grown up 
around the notion of performance indicators, so that all 
manner of powers and virtues are ascribed to then: and 
expectations are aroused that b)' collecting and using them 
great benefits will miraculously result. This eironeous atti 
tude arLws when performance indicators are considered 
in isolation and out of the context in which they should 
properly be //5t'£/(Teather 1990a, p. 103). 

Tlie Netherlands embraced performance indicators and 
quality a.s.sessment primarily as a result of the economic crisis 
in that country during the early 1980s. In re.spon.se to as.ser- 
tions that education and research, which consume approx- 
imately 20 percent of the government's annual budget, are 
highly inefficient, higher education had to put its economic 
hou.se in order to relieve the pressure on the country's expen- 
diture.s. A nev,' managerial s>'.stem using perforniiince indi- 
cators was regarded as a way to bring needed fiscal respon 
siveness and di.scipline to Dutch higher education. As part 
ol the ciesired response, for example, in.stitutions began pro- 
ducing "fact books" containing such information as graduation 
rates, .student, faculty ratios, and personnel profiles (Maassen 
and \'an Vught 19H8; Spee and Bormans 1992). To reward this 
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rcspt)n,si\-encss, a new qualin- control system, \vhifh linked 
increased autonomy with greater quality control, went into 
legal effect in August 1993 (Sander 1993). 

Tlie concept of indicators reached fruition in the United 
Kjngdom b\- the late 1980s. The use of performance indicators 
was highly correlated with the rise of Margaret Tliatcher as 
prime minister and her insistence on government agencies' 
being more responsive to the free market. Thatcher (and oih 
ers, including Ronald Reagan in the United States) indicated 
that the education system should be more self-reliant and 
more dependent on market forces, rather than merely looking 
for ever greater government support without accountability 
(Gt)edegebuure, Maassen, and Westerheijden 1990a; Temple 
and Wliitechurch 1994 ). ^M^ile the Linited Kingdom demanded 
more responsiveness to market priorities and Australia 
demanded more response to government priorities, both 
countries ftx-used alm-ost concurrently on the use of per. 
ft)rmance indicait)rs as a point from which to make their 
academic and managerial itssessments of educational 
performance. 

In the United States, the Wisconsin and State University of 
New ^'ork (SLi>A') sN'stems were rewarded during the late 
1980s and early 1990s with greater autonomy by their respec 
ti\-e state legislatures for responding to legislative concerns 
before polic)' makers flexed their legal muscle (Ruppert 
1994e). In 1988, for example, then Provost Joseph Burke of 
SU>A' launched an initiative fcx'u.sed on assessment that had 
as its major gt)als the imprc^vement (^f teaching, the e\-aluation 
t)f the uni\ersil\''s effecti\-eness, and the measurement of 
learning outcomes. Wiile not mandated by the state, the effort 
has provided convincing evidence to governors, legislators, 
and the general public that .SLW is producing desired results 
without legislation. 

Although the reasons (or their use var\' from country' to 
ct)uniry, the concepts of quality assessment and cjuality indi 
catt)rs and their use as an integral part of planning, alkration, 
and assessment are c|uite similar. 

A key factor in the international emergence of perft)rmance 
indicators is their side effect on the rapid growth and progress 
t)f international technology. LLser-friendly, accessible, prolific, 
and t\)mprehensi\-e data systems were key factors in the emer- 
gence of performance indicators (Lucicr 1992). Wlial was 
available was collected and measured. ^X'hat was measured - 
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i)r measurable- -was given value, and what was given value 
was reviewed for accountability and funding. Just as there is 
or should be a link bers\'een planning and budgeting, the pro- 
liferation of sophisticated information technology-, beginning 
in the 1980s, was linked— or at leiLSt highly correlated— to 
the rise and tise of performance indicators. While the concept 
of indicators arrived slightly earlier in Europe than in the 
United States and has become more widely entrenched there, 
the lack of information technolog\- at the same level of sophis- 
tication as in the United States has made Europe and other 
countries lag somewhat behind some of the more sophisti- 
cated efforts in the United States. 

This .strong tilt toward accountabilit\- and assessment in var- 
ious countries shared some common themes of development 
and implementation with many .states in the United States dur- 
ing the middle to late 1980s. One characteristic of the move 
toward assessment in the United States was "the incredible 
speed of its evolution" ( Ewel! 1993a, p. 12), and by 1989, 15 
.states h.ad some form c>f legislation on measures of student 
outcomes, with 13 ha\ing pending legislation (Sims 1992, 
p. So). Another pattern in the United States that emulated the 
European prciccss was the tendency to use "legislation by fax" 
(Ev.-ell 1993a. p. 12). Imitation became the sincerest form of 
flatter}- in much of the legislation mandating assessment. For 
exaniple. New MexiccVs "report card bill" was parroted in 
South Carolina's Act 225, and Kentuck>-'s SB109 sub.sequently 
emulated South Carolina's act— suggesting a pattern of com- 
munication and commonality in legislation and data items 
collected by the various .states, much the same way that it 
(Kcurred in Europe beaveen countries. 

As a restilt, both mistakes and some factors worth emulating 
were transplanted across states. For example, one conmionly 
used or considered state performance measure or indicator 
(e.g., in Texas, South Carolina. Wisconsin, and Tennessee) 
emphasizes senior or experienced faculty's teaching lower- 
division courses, with the implicit implication that senior 
faculty are "better" teachers. Little evidence based on research 
suggests this assumption is correct, however (Feldman 1983). 

Other .states or systems (e.g.. Colorado, New York, Kentticky, 
and Florida) use indicator s>-stenis that assume lower student/ 
faculty ratios or smaller class sizes are a.s.scKiated with a 
greater cognitive gain in learning (see, e.g., Ewell 1994a, p. 

156). i:)e.spite such o.stensibK- ccimmon conceptions by the 
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public and poliq' makers alike, the literature fails to link small 
cla.ss size or k)wer student/faculty ratios with increased learn 
ing gains in the classrcwm (Dubin and Taveggia 1968; 
McKeachie 1980; Tomlinson 1988; Williams ct al. 1985). True, 
overwhelming research results indicate students belierc they 
learn more and are more satisfied with smaller classes or a 
lower .student/faculty ratio, but little research evidence vali- 
dates these impressions (Ewell 1993b, pp. 16-17). Would 
indicator systems with incentives that enhance learning 
prixiuctivity through more self-paced instruction or greater 
use of technology' be more effective in producing the 
desired results? 

Finally, another common flaw in indicator systems is their 
tendency to reward (or punish) a single in.stitution for stu 
dents' summative performance. Holding degree-granting insti 
tutions totally responsible for the competency of their grad- 
uates is a logical, but mind-boggling, research problem for 
performance indicators. Assessments of the source and qualii>' 
of student learning outcomes in general or of remedial edu 
cation courses, for example, are particularly difficult to deter 
mine in comprehensive colleges, dcxtorate-granting institu 
tions, or research universities. Many, if not mo.st. students in 
such institutions do not take the majority of their general edu 
cation courses at the institution from which they receive their 
degree (Smith 1993). Further, the courses taken by students 
at a community college do not produce the same results as 
those taken at a baccalaureate-granting institution (Ratcliff 
1992, p. 65). Transfer students are generally not .separated 
from native students when summative competencies are eval 
uated. In short, an assumption of a common intellectual cxpe 
rience by graduates of a particular institution is interwoven 
into many indicator s\'.stems, and the validity seems dubious 
of a statewide indicator s>'Stem with incentives and rewards 
tied to the learning gains by .students of a single in.stitution 
and not a collection of experiences. 

And herewith lies a problem with some performance indi 
cators: They often lack validity and reliabilit\-. Tlie tendenc\- 
to imitate other programs sometimes creates a tendency to 
adopt unproven measures or indicators not based on research 
findings. On the positive side, however, such imitation sug 
gests common perceptions of prolilems that extend beyond 
.state borders as well a.s policy makers' willingness to act 
promptly on solutions. 



The Quality Revolution 

The 19H0.S witnessed tlie emergence of ;i spirited public 
debate about tiuality, not only in liiglier education, hut also 
in the corporate sector. The media were filled with criticisms 
of current practice, with calls for improvement, and with par 
ticular concerns about the U.S. market's vulnerability to 
Japanese products (see, e.g., Kenncd\- 1993; Porter 1990; 
Thurow 199.3). Tlie debate proved fni.strating, not only 
bec;iuse the \'oices bec;inie .strident, hut al.so bec;iusc the 
responding nics.s;iges were often contradictoiy. At the s;ime 
time, for ex;iniplc, th;it the Ford Motor Comi)an\- proudly pro 
chiimed "QuiilitN- Is job One." the comp;in\- w;is recilling its 
products ;is ;i result of faulty nianuf;icturing ;ind ;i.s.sembly. 
I'ingers were pointed not only ;it industry' but ;il.so ;it educ;i 
tion— ;ind righlK' .so. Students' m;ith ;ind verlxil and written 
cxpi-.-; :;i. n skills were wc;ik ;icross the n;ition, ;ind .st;indard 
i/.ed test .scores had declined nc;irl\- 80 jioints in the jirevious 
three dec;idcs ;is .spending incrc;ised significantK' (B;irlon 
199^■ Bennett 1993; Murray ;ind Herrn.stein 1992). The per 
ccntage of educ;ition budgets de\-oted to rcmedi;il and dc\cl- 
opnicnt;il cducition w;is ;il.so incre;ising, ;ind the resulting 
qu;ilitv of .students, to the public eye, did not .seem to be 
improving, indeed, if many of lho.se .students h;id been ;iuto 
mobiles, tlie\' likel\- would have been recalled hy the high 
.schools or colleges from which thc\- gnidu;!tcd. Rcmcdi;il or 
de\elopment;il education prognims in the three R's to bring 
high school graduates up to college enir\- .skills bccime com 
nion on college c;impuses b\- the end of the 1980s (Clowes 
1992, p. too ). Public opinion clearly rctleclcd a highly critical 
attitude toward educ;ition;il ni;il|inictice. It was "the llr.st time 
since the kite 19S0s thai the .Amcricitn people [werel .so 
arou.sed ;ibout the ciuality of cduc;ition" (GatY 1991. p. S). In 
view of such widespread rcmedi;ition, it is not suqirising th;it 
college ediK ated Anieric;ins are not \ers- litenite. A summ;in- 
of 2.(iOO studies on higher education outcomes by the Hdu 
c;ition;il 'le.sting Seivice Policy Information Center kaind that: 

• .'\bout h;ilf of college graduates cinnot understand ;i bus 
.schedule. 

• Only 13 percent of college gniduatcscan perform 
multiple .step m;ith iiroblems. 

• OnK 1 1 juMCcnt of four ye;ir graduates ;ind i percent of 
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nvo year graduates can properly summarize a passage on 
how lawyers challenge prospective jurors (Barton 1994 ). 

llie budget does reflect public values and national priorities 
'or higher education (Albright 1985. p. 16; Bogue 1993; Folger 
1984b, p. 1; see also Darling-Hammond 1992). During the 
1960s and 1970s, for example, national goals for higher edu 
cation addressed serial inju.stices and expanded access. Efforts 
U) increase minority- enrollments and financial aid programs, 
with low tuition rates to support increased enrollments, were 
the order of the day. Budget meclianisms. primarily funding 
formulas, were prevalent a.s a means of promoting growth in 
higher education and equity in the allocation of re.sources 
(McKcown 1989; Mo.ss and Gaither 1976; Noe 1986). Input 
\'ariablcs, such as enrollments, were emphasized. And accred- 
itation re\-iews favored such input variables as volumes in the 
librar\- and number of faculty with terminal degrees. 

Bui with the 1980s and the public's percei\-ed erosion of 
cjuality in education, the goals for higher education shifted 
in almcxst epideinic proportions to a focus on quality— by the 
academy and the public alike. From January 1983 to 
December 1990, for example, ERIC lists 13,541 publications 
explicitly referring to "quaiity" in education (Noble, Cr\-ns, 
and [aur\' 1992, p. 310). In 1986, a task force of the National 
Governors As.scK-iation ccincluded that, among other things, 
go\'ernors, .state legislatures, and coordinating boards should 
adju.st input-oriented funding formulas to prcnuie incentives 
for impro\-ing the quality' of undergraduate education, based 
on the results of a comprehensive as.scssment program (Sims 
1992, p. 52). 

National goals for higher education in the 198()s— indeed, 
to the present— remain much the .same as the\- were in the 
1960s and 1970s, with an emphasis on equity, access, and .stu- 
dent financial need (Mingle 1989, p. 8). But even at the fed 
era! le\-el, at lea,st by the late 1990s, performance evaluation 
in the form of peer re\-iews and a.s.se.ssments by external agen- 
cies can be expected to intensify-, lliis trend at the federal 
level toward greater accountability for federal funds will prob 
ably result in uni\ersities' being required io examine their 
priorities and perhaps restrict their organizations to achieve 
greater efficienc\' without compromising effecti\-ene.ss. 

Reforming the prcK'css of accreditation appears to be a cur- 
rent focus as the federal government expands its in\-ol\'ement 
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in performance assessment, In Deeeniher 1993, the C'ouneil 
on Postsecondar\' Accreditation (COl'A), the major member 
ship organization tor accrediting agencies, was disbanded. 
Part H of the Program Integrity Triad of the Reaiitiiorized 
Higi'-.er Kducati(.)n Act of 196S portends growing federal intru 
sion in accreditation, as institutions participating in the .Stu 
dent Financial Aid Program of Title IV are subject to giA-ern 
mental reviews of curriculum, student personnel policies, and 
quality assurances, onc e the pur\'icw of accrediting agencies 
and institutions. In sum, the federal go\'(?rnment appears on 
the verge of no longer leaving e\'idence of tjuality and licen 
sure to state and accrediting agencies (,Sims 1992), suggesting 
a growing federalization of higher educaticMi- a common phe 
nomcnon in Europe, but a radical shift in I'.S. educational 
poliq'. Particularly for private and independent institutions, 
this shift repicsents a dangerous intrusion into campus opor 
ations and could precipitate th'' homogenization of higher 
education in the I'nited States. Indeed, the mo\'ement toward 
state a,ssessment was partly a response to national concerns 
about ([uality. but it migiit be too little too late. Higher edu 
cation ostensibly mu.st prepare for a three tier s7,stem of 
review that includes national ba.seline .standards, state per 
formance standards, and accreditation .standards for recog- 
nition of overall excellence. And reports of such findings are 
likely to become public, communicating iastitutional .strengths 
and weaknesses to a broad segment of the public- -and acting 
as an incentive for change or as a rallying point to cn'.stallize 
instituticMial opposition to external re\'ievv. 

Another national etTort, although still largely .symbolic, is 
expansion of the Malcolm Baldrige National Quality Award, 
handed out annually by tiie U.S. Department of Commerce, 
to inc lude .schools, colleges, and hospitals. Plans are to ha\'e 
a pilot conte.st in 1995, with no winners announced, and a 
full contest in 1996 (I-uch.sberg 1993a). Because measures 
of quality in education and health care are not as well defined 
as they are in business, this award and the criteria to be cstab 
lisiied arc sure to be controversial. Questions are also being 
raised as to whether such a "quality contest" creates the right 
incenti\'es for change, and concerns are being voiced that the 
criteria, once established, could become a set of national per 
formance .standards for «// educational in.stitiitions. A national 
indicator s\',stem could be in our future as a result of this cli 
mate fc)r a fuller accounting of how well iastitutions of higher 
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education arc preparing! 'Av.nr stuilcnts and usini.', tlicir rcsour 
CCS (rklgcrton 1991 ). 

i'roni the pcrspcciicc ol'liic states, iiowcvcr, liic educational 
aj^enda in the 1980s iiad already shitted to such issues as 
assessment, more sclecti\'c admissions and stratification 
among institutions, the core curriculum, belter undergraduate 
education, merit .scholarships, incentive funding and a 
greater emphasis on tiuality, "The elixir ol'the 198()s was 
quality" (Mingle 19H9. p, 8). 

In the 1960s, funding IbrJiiulas were the budget veiiicle 
the .states used lo jironiote growth, and access, while perfor 
mance or incenliw funding, which emerged in the 1980s, w.is 
generalK' designed to sup/>lci>u-Ht the still widely existing for 
mula allocation .s\-.stem and lo promote greater cfTcctivencss. 
I'lirtlier, the formula .system was de^signed iirimarily around 
enrollment driven input or acti\ ity measures, wiiile perfor 
niance and incentive funding generally encouraged institu 
tions lo measure initconH's, a much more slippeiy and ab.stract 
activity that has proved elusive, difficult to define, multifa 
ceted, and much more i"Xpensi\'e and uncertain for planners 
to measure. !n Florida, for example, higher education officials 
estimated that two .systemwide reports required by a 1991 
"accountability" law would cost $1..^ million to develop 
through new software and .student e\'aluation forms. I'iorida 
officials .say they cannot afford such additional expenditures 
when budgets for course offerings and libraries are being cut. 
In South Carolina, institutional research directors estimated 
it would co.st $20,000 for each institution to compile a "report 
card" that measures the pertormance of higher education 
(Blumenstyk 199.-^). 

Thus, these outcomes and prescriptions based on quality 
often come at a prii'c. external agencies' greater intrusion in 
academic issues and institutions" ab.ioipt'on of new i o.sts. 
Faculty and administrators alike shv '.'.d ■■emember that this 
external concern for accountability in state capiitals and in 
Washington has emerged, in part, becau.se of the academy's 
piartial default on its pi"ofe.s.sional obligations to produce the 
desired results and inform the public and students. 

The Productivity Chiallenge 

M qualit\- became a national concern in the 1980s, the word 
took on political implication.s among policy makers. Hut along 
with the public's and .state leaders" rising concern aboia qual 
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ity canu- a concomitant concern about cost and productivit\' 
in higlic." ■ 'ucation (Hollins 1992). Tliesc same extornal pol- 
icy groups often included the concept of "prcKluctivity" under 
tlie umbrella of quality (Mingle 1989). This connection and 
the v(K-abulaiy itself are largely anathema to many faculty, 
hoivever, who view .suspiciously the unwelcome t)verlap 
I'jetween the values of higher education and business— a dis 
tinction between the service and the profit sectt)rs. 

The twin issues of cjuality and productivity have been 
interrelated since the 1980s, and the concern appears to be 
.strengthening among state leaders who seek concrete results 
and indicators of "scn'ice to the .state." To many .state leaders, 
a quality in.stitutit)n is a prt)ductivc in.stitution, prt)viding 
mcasLirab'e returns for the state's economic aid in the form 
of researcf. dollars, training, and seivicc. A study of the "^LiW 
.system (Slaughter 1985) confirms that faculty'.s expected role 
has shifted sMice 1970 from primarily seiving students to .sen.' 
ing the .state's cc(MK)my. Faculty in the study were expected 
to be more productive, for example, by linking their research 
to indu.strial revitali/.ation, to train .students for "high tech, 
high co.st, high-return" jobs. In the main, faculty are being 
asked to be more "productive" to help .states overcome their 
fiscal problems by revitalizing their industrial ba.ses. Perfor 
mance indicators t(^ measure an in.stitution's contribution to 
a state's economic goals are becoming more ctjmmon (see, 
e.g., Ashworth 1994, p. H; Borden and Banta 1994; Ewell 
199-fa,p. 1S7). 

But the broader discii.ssions abtxit what represents a pro 
ductivc in.stitution are h:iz)'and fragmented (Hodgkin.st)n 
1981 ). Current di.scu.ssions tend to take at lea.st two distinct 
perspectives. One group tends to fcK'us more on iastitLitional 
ejjk lency -dnti a more qiiantit(itiivdppioM:h in which pro 
ductivity means working harder and fa.ster and producing 
nK)re measurable units of what is produced. An increa.se ■! 
faculty credit hours or research dollars over the previous year, 
for example, would be an acceptable increase in productivity 
that %votild include measures or performance indicators to 
evaluate the desired results. Tlie public and other con.stituen- 
cies beyond the campus prefer this viev^point. 

A .sect)nd perspective on productivity is more concerned 
with I'ffi'ctitvut'ss -dnd continuous impnnvmcnt in quality, 
for while "efllciencA' ckics not include effectivene.ss, . . . eftcc 
ti\-ene,ss docs include efficiency" (Burke 199.^b, p. 4). Within 
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the world of higher education, this definition is the more pal- 
atable one. Measures like the amount of published, refereed 
research produced, enhanced prestige among peers, improved 
college ratings and rankings, greater amounts of resources, 
and an increase in the number of citation indices, for exam- 
ple, represent a "productive" institution (Bogue and Saunders 
1992, p. 65; Cave, Haimey, and Kogan 1991, pp. 128 -35; EiLsh 
1983; Jalongo 1985). Clearly, these two viewpoints about pro- 
ductivity ovedap, but one theme or the other seems more 
dominant with certain constituencies. 

For example, concurrent with the concern about the qualit\' 
of undergraduate education is a wide.spread belief that faculty 
workloads are declining and that the time sju^nt by fLiU-time 
faculty in the classroom is declining Oacob.son 1992; Jordan 
and Uyy-ell 1992; Ru.ssell 1992). .\ .study of faculty workload 
in 1993 conducted by the Oliio Hoard of Regents found that 
teaching loads had dropped 10 percent in the previous decade 
in state institutions. Tlie result was legislation to increase the 
amount of time faculty devoted to undergraduate education 
(Magner 1994). In Texas, the Coordinating Board for Higher 
Education developed and the legislature ackipted as one of 
its performance indicators a .semestcr by-.semester report to 
the Ivcgislative Budget Board about the percentage of time 
full-time faculty .spend with lower di\'ision courses. 

Paradoxically, as faculty workloads o.stensibl\- declined, the 
workload of the "average" American was increasing. Toda\- 
the a\'erage worker works one extra month per \ ear compared 
to 20 years ago (Schoor 1991 ). /Vnd the increased workload 
has been accompanied by an increase in the number of peo- 
ple working — a nece.ssity to maintain comparable .standards 
of living. From 1972 to 1992, the number of two earner house- 
holds rose 41 percent, the number of single-parent wage 
earners 156 percent (Schoor 1991 ). 'Iliis increased workload 
in the private sector could help explain the public's and policy 
makers' growing .scrutin\- and criticism of the neglect of teach- 
ing in favor of research. 

The charge that some faculty are unproductive, inefficient, 
and incompetent becomes more tenable when some data are 
examined. A national study of workload conducted by the 
Carnegie Foundation for the Advancement of Teaching con 
eludes that "the a\'erage faculty member devotes about 20 
hours a week to the classroom" but spends only "an a\'crage 
of 9.8 to 10.5 hours a week />;the cla.s.sroom." Further, 45 per 
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cent of the nation's faculty "had no professional writings 
accepted or published during a given nvo-year period" (Mac 
roff 1993). Although such statistics might be sufficient to label 
faculty "the new leisure cla^ss" (Maeroff 1993). not mam- 
inside the academy would agree (Magner 1994. p. A18); 
nevertheless, as of .^pril 1992 "at least a dozen states" were 
seeking performance data on faculty work weeks and contact 
hours (J^«''">-'^"n 1992). 

If indi\'iduals respond to financial incentives, then the rca 
sons are clear for undergraduate instructions suffering. A 
study of o\-er 4.000 tenure-track faculty at a wide range of in.sti- 
tutions reached the folkwing conclusions; 

• 'Hie more time \-ou spend on teaching, the less 
compensation. 

• The more hours in cla.ss per week, the lower the pa\'. 

• The greater the time .spent on research, the higher the 
compensation. 

• Faculty who teach only graduate .students get paid mo.st. 

• Tlie greater the number of refereed publications, the 
greater the income (Jacob.son 1992). 

llie per\asi\-ene.ss of the incentive .sy.stem can be aptly illus 
trated by a review of the criteria for the Sl^^A' Di.stinguished 
Teaching Profes.sorships, which are intended to reward 'he 
.SN'.stems be.st teachers. The criteria require— and the .s\-.stem 
wide committee rigorousK' enforces -"a sub.stantial recorti 
in research and publication" (Burke 1993b. p- ""). In contra.st. 
the Di.stinguished I'rofes.sorship. which recjuires a sub.stantial 
national record in publications and research, requires no e\i 
dence of good teaching skills (p. 7). Clearly, mo.st facult\' who 
do not .stress the primary- role of research o\-er teaching will 
not rise to the top of their profession or receive recognition. 
But "what is truly amazing is that there are so many facult\' 
in research universities who care about teaching despite the 
virtual ab.sence of rewards for doing so ' (Edgerton 1993a. 
p. 16). 

A proposed faculty incentj\-e and reward .sy.stem would have 
tenured faculty guaranteed a threshold ,salar>' and earning the 
rest of their .salaries each year as rewards for achic\-ing agreed 
upon goals (Heydinger and Simsek 1992). Such a .s\-.stem is 
intended to counter the unintended but ob\ ious wa\ s thai 
curreiit incentives [produce prevailing practices. 
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Analyses of performance and productivity in several studies 
conclude that much of the reason productivity has dcclmed 
stems from facult)' members' shift toward research and pro- 
fessional activirv' and away from undergraduate teaching, as 
prestige in the academic culture comes from research rather 
than teaching. Although in one study "educational qualirs'" 
was improved initially as a result of the instructor's research, 
it subsequently declined, and the final results were an in- 
crease in the cost per student, less qualirs' time for under- 
graduates, and a proliferation of support ser\'ices. And the rela- 
tionship drove up costs in academic departments (see, e.g., 
Hollins 1992; Mass)' and Meyerson 1994; Massy and Zemsky 
1990). Spurred on b)' such evidence, public policy makers 
are .seeking more public accountability, particularly in relation 
to undergraduate education. 

One of the more innovative campus or system respon.ses 
to the challenge of productivity' comes from SLJNY. Despite 
the need to increase productivity, advances cannot be made 
by "reducing or cheapening the inputs" (e.g., cutting faculty 
or .staff) or increasing the output (Johastone 1993a, 1993b). 
Any "significant and sustainable" increa.ses in produciivirs' 
must come through the learner's productivity; that i.s, students' 
"demonstrated niiistery of a defined body of knowledge or 
skills," for example, by using a .set of prescribed measures 
or indicators for individually paced learning (p. 2). The key 
ingredients for such a s>'.stem are ( 1 ) a set of clear, compre- 
hensive, and measurable learning objectives; (2) a.s.sessment 
instruments; and (3) moving the student on to a new set of 
objectives after ma.stery and measurement of the initial ;Lssign- 
ment Qohnstone 1993a, 1993b). 

Accompanying this concept is SLIN\'s development of a 
s>'stemwide set of performance indicators (see table 1 ). A 
common .set of macro indicators has been developed for insti- 
tutions in the system, and individual campu.ses are develop- 
ing micro indicators that reflect their in.<;titLiti()nal missions 
(Richard.son 19941"). Along with this system is an accelerated 
degree program that went into effect at seven of the 64 SL>NY 
campuses during summer 1994. A number of other institutions 
have followed suit with offerings ot three-year degrees that 
respond more to economics, such as making college more 
afft)rdable for .students, than to any demand for productive 
learning (Stecklow 1994). This situation is more an anomaly 
than a common [lattem, however, as delay of graduation until 
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TABLE 1 



SOME PERFORMANCE INDICATORS FROM THE 
SUNY PERFORMANCE REPORTS 



' Funding Lontt'Xi 



• A(.a'.s,s 10 iindcrgraduatu 
(.•dilation 
-Admissions 



-Enrollnicni 



Indicator 

1. Revenue hy soiiive 

1. Tiiiiion rate trends 



Comparison 

1. Trend states 

2. Regional national 



3. Educational and general 3 Prev ious years: na 
expenditures per student tional. peers, suite 

4. Benchmark Svsteni SI '>r>' sectors 



■\|iplkanis. :icce|H.in(. es 

Total, race ethnicitv, level 
age 

.Assessment plans anil 



• rndi'rgradiuiie quaiitv 
( sindenis and institutions i pass rates 
Hisic .skilb 
Cla.ss size 

Xoir A more exlen.si\e list ( )l' SI 'Vi '.s mials and n.'lated indicators ean he found in Burke IW.-^a. pp. .-^l y 



Trends in cnrt)lled 
accepted and race 
ethnicity 

Previous years, state sec 
tors high schiK)l gra^l 
uation trends 
Prev ious years 

PericKlic reports 
Trends 



Data Source 

1. IPEDS; institutional 
research 

2. NASL1GC AASCr 
sur\'eys 

3. IPEDS finance sur\ey; 
in.stitutional research 

-I. Finance and business; 
institutional research 



In.stitutional research 

institutionai research 

In.stitutional research 
In.stitutional research 



after five or .six years of attendance is not iiinisual because 
of .students' inability to take the needed courses or the neces- 
sity to work to offset mmg costs (Dodge 1991 )• 

Such debates about cjuality and productivity liave become 
common in the majority of states. And this dichotomous per- 
spective about productivity has framed a fundamental debate 
between pt)lic>' makers and academicians over public account- 
ability for higher education and the ti.se of performance 
indicators. 

Summary 

Tlie 1980s \vitne,s.sed several attempts to breathe life into the 
idea of greater accountability for higher education, with the 
main .stimulus for such reforms externally imposed and po 
litically based, lliis drama unfolded through a number of 
themes and i.ssues-- input versus output, excellence versus 



Mi'cixKn'iig I '[> 



(-luality. cost versus productivity — all of them accompanied 
by a concern for the qualit\' of undergraduate instruction. 

For example, the earlier concept of excellence centered 
primarily around the accumtilation of resources and an in.sti 
tutions reputation (Burke 199'4 ). Excellence in a college or 
uni\'ersity depended largely on such inpi.it \ariables a.s 
number of faculty publications and grants, students" SAT 
.scores, and the comprehensiveness of their degree programs. 
One problem with such a .striictured. narrow definition of an 
in.stitution. howe\'er. is that it is primarily .static in nature. It 
tends to review the condition of a college or university at dis 
parate points in time and adds little to monitoring t\ hat an 
in.stitution produces in terms of tiutpiits and outcomes - that 
is. results— using such resources. Tlie sub.sec]uent emerging 
notion of quality fcx'used on a more dynamic, more expansive 
notion of goal-oriented in.stiti.itit)nal excellence ba.scd on per 
formance and continuous imprownient. on desired results 
rather than resources, or on "missitin fulfillment" (Bogue and 
Saunders 1992. p. 2''9). 

Performance indicators emerged partly in response to pres- 
sures for colleges and uni\'ersities to demon.strate desired 
results for restiurces prtivided. This subtle shift from excel 
Icnce to a new notion of quality, which also incorporated 
a.sse.ssment. helped meet the public's and pt)li(.y makers' 
demands for more accountability. Yet the transformational 
asix'cts of this new notion of quality have failed to inspire 
or motivate faculty to unite behind a common purpo.sc to 
ensure achievement of an in.stitution's full potential (Burke 
1994). 

The .speed of these changes and their international jxirallels 
are .striking, and keeping these interrelated themes separate 
is probably an impo.ssible task. 0\'erriding all of the.se con- 
cepts is the central idea of greater accountability and achieve 
ment of mi.ssitin iind goals, accompanied by disparate meth- 
ods of incentives and rewards— but with a concentration on 
evaluation and a.sse.ssment using indicators to seek and meas- 
ure performance. 

Shifts in .sentiment are ob\'ious. howewr. emphasizing in.sti 
tutional effectiveness and .student outcomes (Bogue and 
Saunders 1992. p. 160). with "the owrwhelming public pos 
ture for .state a.s.se.ssnient . . . ■impro\'emeni," ik)1 'acct)unta 
bility"" (Ewell 1990. p. 3)- Indeed, the "emerging jiolicy con 
scnsus" anu)ng .state leaders wa.s to act luii still retain a will 
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ingness "to allow institutions full discretion in such matters" 
(p. 3). 

State policy issues for tiie 1990s include: 

• Tlie assessment of educational performance and 
outcomes; 

• The development of new measures of accountability; 

• The improvement of productivit\' for b()th management 
and educators; 

• Tine refcxnis and revision of campus missions (Boguc 
1993, p. 3). 

Accompanying these themes is a nuwement toward greater 
centralization of authority (some w()uld say interference) by 
the states and in particular the federal government, and a ten 
dency toward less instituti()nal discretion. Greater public ac 
countability will become increasingly important for the rest 
of the decade. 

Inf()rniati()n technologv' and government's greater inxolve- 
ment and c()ntrol over education have enhanced the need 
for and allowed the creation of a refined .system ()f perfor- 
mance indicators (Cave, Hanney, and K()gan 1991. p. i"^)- The 
remainder of this book maps the emerging contours of this 
reform mcn'ement and details .some of the maj()r domestic 
and internaticmal accomplishments to date as well a.s emerg 
ing trends. 
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PERFORMANCE INDICATORS IN THE UNITED STATES 



A number of new players in the United States have fashioned 
broader, more diverse clusters of performance indicator s>'s- 
tems. In addition to the SUNY and the University of Wisconsin 
systems' projects, previously mentioned, other broader- b;ised 
models have appeared from which national, slate, system, or 
institutional efforts might be constructed. These efforts con- 
stitute a useful reference point for reviewing indicator systems, 
present or proposed. Although many indicator systems have 
some potential flaws, particularly with reference to the use 
of indicators that lack validity and reliability, individual cam- 
pu.ses can still profit from these efforts. Indeed, "a set of 
national benchmarks about key experiences and conditions 
in general education . . . might be of considerable value in 
determining the degree to which colleges and universities 
are able to act consistently with . . . national goals" (Ewell 
1993b, p. 5). Wliile "all organizations make mistakes at first" 
in the initial development of a performance indicator .s^'stem, 
the "adoption of crude performance measures" and their 
refinement into the ultimate indicator system are the goals 
(Osborne and Gaebler 1993, p. 156). "Often simple is best, 
even if It initially seems less technically attractive" in devel- 
oping indicators (Ewell and Jones 1994, p. 16). An indicator 
system mast be flexible to permit evaluation of different kinds 
of institutions, using available data where possible to make 
the prcx-e.ss more co.st- effective. Multiple indicators are neces 
.sar>' to capture adequately the complexity of institutional per- 
formance. Finally, the designers of the system must remember 
that indicators are simply that— wc/Zcators of success, not pre- 
cise measurements —and must be continuously improved to 
approximate a .stated goal. 

Indeed, indicators as a .system are best used to check a 
unit's health and progre.s.s. These ,sy.stems, when combined 
with other efforts like accreditation, program review, and peer 
review can become a respected form of monitoring quality'. 
A gocxl performance indicator system can best be used as one 
of man\' in.struments to monitor quality assurance, enhance- 
ment, and effectiveness. But performance Indicator models 
have additional uses bevond this role. 



"Often simple 
is best, even 
if it initially 
seems less 
technically 
attractive" in 
developing 
indicators. 



The Use of Performance Indicators 

It is appropriate at thL<i point to discuss in more detail the uses 
and shortcomings of performance indicators. Tliis di.scu.ssion 
proposes a .set of principles that are essential in the design 
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and implementation of effective pcrtbrmancc indicators and 
points to some of the uses, benefits, and burdens that can be 
derived from that effort. 

Tlie definition of performance indicators includes .several 
descriptors: effectiveness, ef^^iciencA'. i,nputs. prcx'ess. outputs, 
excellence, quality-, accountability, and others. Indicators are: 

. . . a coiicrett' piece oj iiijormation ciboiit a conch'lioii ur 
result of public action that is regularly produced, publicly 
reported, and systematically used for plauuitig nionitoring, 
or resource allocation at the state or s}'ste»i level . . . [They 
arej intended to he used together, not singly or out of con- 
text {Ev>-e\\ and Jones 1994, p. 7 ). 

. . . an authoritative measure — usually in (/uantitativefomi 
. . . (Cave. Hanney, and Kogan 1991. p. 24). 

. . . ratios, percentages, or other (/uantitative values that 
allow an institution to compare its position in kiy strategic 
areas to peers, to pMt performance, or to prei'iously set goak 
(Taylor, Meyer.son. and Mas.s\- 1993. p. x). 

Tliesc definitions correctly emphasize performance, but 
another theme that frequently occurs in definitions or dis 
cussions of performance indicators is the quantitative miura 
of the descriptors. Tlie ii.se of such data, however, suggests 
a precision of measurement that is often unattainable; as the 
label suggests, they are indicators of performance, "signals 
or guides rather than ab.solute measures" (Sizer, .Spee, and 
Bormans 1992, p. 13"^). Tluis. multiple indicators are neces.sar\- 
to produce a more accurate picture of an in.stitution's ox'erall 
operation, but valid reliable miWcAoK arc also necessary. 
Bad data are worse than no data when indicators become the 
primars' tool for managing higher education or alkK'ating 
resources. 

Performance indicators have five primary uses: monitoring, 
ex'aluation. dialogue, rationalization, and allocation of resour- 
ces (Sizer, Spee, and Bormans 1992). 

1. A/o;/;Vo'7>/g. A stud\' of (iiiantitatixe management ap 
proachcs to higher education nearly two decades ago 
noted that "the paramount need'" was for "a .st.andardized 
and consistent ciiiantitative language" to monitor the use 
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of sucli tecliniquc'S (biwrcncc and Scn-icc l'^", pp. 
25). Tliut language lias inw largely been established and 
the teehnologN- de\ek)ped so that data, siieh as multiple 
performanee indieators, ean be eolleeted and combined 
tc- eharaeteri/.e a partieular institutional environment. Sueh 
efforts highh' depend on the existence of in.fonnation s\s 
tems that, in turn, should dovetail with decision making 
and the institution's or system's responsibilities. The pres 
cnce of accurate and reliable data now allows tl ic mon 
itoring of developments in a sv.stem, state, or nation. 

2. Eraltiation. Evaluation focuses on the attainnienl of goals, 
and tibjectives are normally quantifiable to enhance or 
facilitate such measurement. Multiple multidimensional 
indicators are necessaiy, cvlmi for the formatiw e\aluatit)n 
of an institution (Rogue and .Saunder.s 1992): in their total 
ity, such indicators would help pro\ ide a rational ba.sis 
for decision making about progre.s.s toward or attainment 
of goals. 

3. puiluguc. I'sing indicator .sy.stem.s, two or more panics 
are able to communicate with each other, attaching the 
.same meaning to abstract conceits. The problem.s inher 
ent in the lack of a standardized language haw largeK' 
been solved thrcxigh such agencies a.s the Organization 
for Economic Cooperation and Development ( OliCD) 
and the National Center for Education Statistics (NCES). 
,\nd using indicators as a point of reference allows judg 
ments, compari.sons. and communication about the eclu 
catkinal t)bjectives of an iastitution. a .state, a countiy. or 
a continent. 

■4. Rcttiuualizcttioii. Rationalization is the accomplishment 
of a coherent policymaking process. Performance indi 
cators can pla\- a crucial role in iilanning. but an effective 
planning prcK'ess must be in place before the application 
of indicators so that the accompli.shment of polity can 
be measured. 

5. The allocation of n'somrcs. Performance indicators can 
be used in the allocation of resources, but their validity 
should be high to prevent any unwanted or undesirable 
side effects, such as pro\ iding funding for indicators that 
ck) not measure what the\- purport lo measure. Becairse 
some I'.S. indicator sy.stems sometimes lack \alidity and 
reliability, .setting norms for comparison cm be lime 
consuming. 




Tlie n.sc of performanco indiaitors ultimately in\'()l\-e.s com 
parisons -against oneself, a nomi, or others, liie use of sueh 
explicit comparisons, absent of judgment, emphatically means 
the identification of in.stitutions or systems as abo\'c or below 
par. liibeling .someone or something a "winner" or "lo.ser," 
perhaps indirectly or unintentionally, is apt to otrur, and uni 
\-ersit>' personnel, ixirticularly faculty, \-c)ice considerable 
apprehension about a .system that employs such summati\-e 
judgments openK' in. sa\-. legislati\-e hearings or .state coor- 
dinating board meetings. Further, when such indicator .s-\'s- 
terns are considered for redi.stribution of existing re.sources 
rather than the addition of resources, the resLilts can also be 
negati\c and intimidating. "VClien organizations face condi 
tion.s they define as threatening, the tendency is to become 
rigid . . . centralized. con.ser\-ati\e, protecti\'e, inflexible, and 
nonadaptable"— with the result often "a .short-term, threat 
induced crisis mentalit\" and the likelihood that organiza 
tional performance will suffer (Cameron and T.schirhart 1992, 
p. HH). Tlic potential for damage with such a .s\stem is thus 
implicit, and .some legitimate concerns ha\'e ari.sen thai focus 
on the use of indicators and the wa\\s in which individuals 
.should be j-jarticipating in their negotiation and dex'clopment. 
If anything is to be accomplished in higher education, '"it is 
going to be done by the j-jrofcssors or it is not going to be 
done at all" (Rudolph 198-1, p. 1.^ ). It follows that any suc- 
cessful, su.stained u.se of i-)erfbnnance indicators in teaching 
or learning in I '.,S. higher education mast inxolve faculty in 
the [-jreparation and u.se of the s\'.stem and in its reward mech 
anisms. I-"urther, the careful de\-elopment of such ,s\',stems is 
crucial becau.se of the potential for irrexocable damage (or 
improN-cnxMit) they offer the organization (Blank 199.^). 

But just how ea.sy is it to de\ eloi^ and implement a perfor 
mance indicator .system that actually worLs? If the experience 
i)f the r..S. business sector is indicative, the an.swer is "not 
ver\'," Tola! ijuality management, benchmarking, reengineer 
ing. empowerment of the individual, and other current renie 
dies are being projiosed to unshackle ( '.S. workers and allow 
their creative juices to flow. But "no empirical evidence 
Ishows) that any of these things [increa.se] productivity"' 
( Bleakley 199.V. see al.so Gar\'er and i.ucore 1993 )- 

Tliis conclusion .should prod public policy makers to reflect 
on what their methods, including performance indicators, are 
doing to .spur or .stifle— cieati\-ity, initiative, and learning. 
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Tlicsf tlirec items should be tlic goals of the remedies legis 
latures and eoordiiiatiiig boards arc using to breathe new 
effcetiveness, eftleicnty, and prodiietivity into their higher 
edtieation systems. 

Since the 195()s, conventional employee management the 
on' has held that people are motivated by the intrinsic satis 
taction of a job: the opportunity for achievement, challenge, 
contributions, excitement, and personal recognition. But tbe 
basic structure of state incentive funding systems -like, tor 
example, the Tennessee Performance Funding Program to 
impioxe instruction, the now defunct New jersey Challenge 
and Compeliti\'e Grants Program, Ohio's Selective Excellence 
Program, and Colorado's Incentive I'inancing Program--- re 
wards and recognizes group or institutional incentiws. often 
using institutional performance indicators to measure results 
before producing such rewards. Do such "collectiv e rewards" 
effectix'elv motix'ate indiv'idual facult\- and students to achie\'e 
creativity, initiaii\'e. and learning'^ The answer to this question 
is not vet known or currently a\"ailable in the literature. 

If the idea, for example, is to encourage individual faculty 
to-be entrepreneurial as a wa\' to b.elp fulfill a state's economic 
goals or to prox ide better-ciuality undergraduate education, 
then int/iriclual imvlvenwnt is neccssarx' for such indicator 
or incenti\'e iirograms to he successful. Rewards must address 
the wants and needs of the individual learner and producer 
I'aiher than fcKUs on a uniform reward or incentix'es for an 
entire organization (Diamond and Adam PW). How the iasti 
tution internally uses these- grouj-) incenti\-e rewards, once 
received, to motix'ate individual learning, creativity, and initia 
live is another nccessar\- ingredient to implement a quality 
and instructi(/nal improvement program that works. 

Asvstein can produce "an innovative organization" 
(Toombs and Ticney 1091. p. "9). Financial incentives for 
summer supplements and necessaiA' resources to investigate 
a panicular effort, for example, are tangible recognitions by 
administrators of what is \'aluecl and what is rewarded. Tlie 
use of "potent .symbols " is valuable in bringing about such 
organizational change (p. ''9). I'or example, a reduction in 
course load to develop an undergraduate seminar would sug 
gest the importance of undergraduate in.struction to faculty. 
/\nd leadens' efforts to communicate in writing and verbalize 
their appreciation and approval (or disapproval) of some 
effort is an essential ingredient for bringing about organiza 
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tional change. "A centra! theme of much of the literature on 
organizational change is that innovating organizations must 
be specifically designed to innovate" (p. 74). Clearly, design- 
ing a rev.'ard .system that includes individual incentives 
encourages inno\'ati\'e behavior that is measurable. 

Wliile it is not easy to implement a quality and incenti\e 
program with performance indicators that actually work, a key 
is to develop a program based on incentives that are mean- 
ingful to individuals, not solely to an institution. 

Any discussion of the introduction of performance indi- 
cators into higher education in the United States mu.st first 
take into account several landmark pieces of international 
literature (see especially Kells 1993; see also Bottani and Del- 
fau 1990 and Cucnin 1987). Other works (Cave, Hanncy. and 
Kogan 1991; Kells i992b). published roughly concurrently 
with se\'eral studies published in the United States, have 
altered the landscape on the subject, both domestically and 
internationally. Further, NCES is nuning toward collecting 
and publishing, for the first time, indicators that will allow 
stales in this countrv' to compare state level performance with 
members of OECD (National Center for Education 1994a). 
It seems inevitable that such reports in the ftiturc will move 
toward more comparisons of .states and countries, just as the 
current thru.st in this countiy compares institutions and state 
support for education in areas like faculty salaries and edu 
cational and general ftintling. Wliile this volume concentrates 
on public four-year institutions, systems, and state efforts, 
many of the principles and techniques found in pri\'ate and 
community colleges are also applicable to other institutions 
(see, e.g., American Asscx'iation of Community 1994; Doucette 
and Hughes 1993; -Sapp 1993; .Sapp andTemares 1991 ) 

The following paragraphs de.scnbe some initial attempts 
toward performance indicators in the L'nited States. 

The NACUBO benchmarking project 

An initial effort to prepare performance indicators, designed 
to create a national data base on higher education, w;ls cievel 
oped by the National Asscxriation of College and University 
Business Officers (NACITSO) in collaboration with the ac 
counting firm Coopers ik Lyhrand. which largely developed 
the project and compiled the data. The project's primary fcKus 
was on developing comparative ccxsts for a variety of admin 
istrative and financial h.incti()ns (see table 2) and creating a 



set of na'.icinal stapdatds or indicaiors to allow participating 
higher etlucation institutions to measure or compare; their 
efficiciity and productivity-. A national "benchmarking" project 
to allow comparisons in 38 functional areas, it was expected 
to allow -or prod- institutions to adapt inno\-ati\'e cost 
cutting tcchnicjues from other comparable universities whose 
performance is more efficient. Several benchmarks from 
industries and businesses outside higher education, such, as 
airlines and communications companies, will al.so be included 
in the project's results. 



TABLE 2 

FUNCnONAI. AREAS IN NACUBO's NATIONAL 
BENCHMARKING PROJECT 



Acatk'inic aiVairs 
Accounts payabif 
Adiiii.ssions 
AJiinini relations 
Bookstore 

Cart-er planning and piacenient 
center 

Central budget dcpartmeni 

Central .stores 

Collections 

Development office 

Knvironmcntal health and .safety 

Facilities 

Financial aid 

l-'ood ser\ ices 

General accounting 

Human resources: Benefits 

atiniinistration 
Human resources: CkMieial 
Human resources: Hiring 
Information teclinolog\' 



Intercollegiate athletics 

Intramural and reocaiional sports 

Ix'gal affairs 

Libran- 

Mail 

Multicampus system 

atlmini.stration 
l^arking 
Payroll 

Police security 

Purchasing 

Registrar 

Sponsored projects 

Student accounts receivable Siu 

dent billing 
Student affairs 
Student counseling 
.•student health services 
Student hou.sing 
Telecomniunications 
TreasuiT: Cash management 
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NAQ'BO's and its participating insiitutitins' approach is 
somewhat of a re\-cr,sal of the traditional rea.sons for collecting 
such data— from using results to ju.stify more dt)llars from 
states, donors, and students to using them to cut costs and 
fcx'iis on productivity (Blumen.styk 1993, p. AhI; Massy and 
Me\'er.son 1994, p. 94). Traditionally, in.stitutions have fcK'Used 



on inpiiis and costs, eciuating better quality with greater 
expenditures. In contrast, benchmarking t(x:u,se,s on outputs 
and the cjualiry of service.s. If an in.stitution improves its .ser 
vices as a result of such effort.s. it i.s in a better position to 
respond to its critics and better endure tne harsh glare of pub- 
lic .scrutiny. 

With an initial .sample of 120 in.stitulions, NACL.BO's bench 
marking project began in 1992-93 with a rwo year trial. One 
of the project's .severe limitations for nonpariicipants is the 
lack of information on ip.di\'idual colleges, as a result of par 
licipants" .sensitivity about result.s. Another limitation is the 
lack of benchmarks for academic departments, although their 
inclusion is now being considered. 

The Peterson 's/AGB strategic indicators survey 

This effort, like NACll^O's project, is designed to provide 
benchmarking for performance a.s.scssment. The project began 
in 199 1 and subsequently was published by the /ViscX-Mation 
of Governing Boards of Universities and Colleges (AGB) in 
two books on strategic indicators (Taylor, Meyer.son. and 
.\la.s.sy 199.3; Taylor et al. 1991 ). An earlier volume by the AGB 
( Frances et al. 19H"') developed a .s-\'.steni of indicators for trus 
tees. 7'hai effort was subsequently e.\panded in a r\'c\' of 
o\'er "^00 colleges and universities to develop comparative 
institutional data (see table .3") that was |-)ubli.shed b\' Peter 
.son's Guides in Princeton ( Ma.s.sy and Meyer.son 1994). Tlie 
survey was "the most comprehensive effort c)f its kind ever 
undertaken in higher education" (p. ). and the authors 
intend to update it appro.xiniately every two \-ears and to 
de\elop data on trends. Results are provided .separately for 
public and pri\'ate uni\-ersities. with brief .statements of sig- 
nificance and interpretation for each .strategic indicator. To 
assist in making meaningful compari.sons, the data are broken 
down according to two-year colleges, regional colleges and 
universities, and research and land grant universities. 

The NCHEMS indicators of good practice project 

This project, for the National Cienter for Higher Education 
Management .Sy.stems (NCHEMS) under the guidance of Dr. 
Peter Hwell. fcxaises on develf)ping a range of "good practice 
indicators" in undergraduate education. As policy, it is tic d 
to the federal effort to de\'elop indicators for tracking progress 
on national education goals, particularly goal 5.S, which 



TABLE 3 



MAJOR HEADINGS IN THE PETERSON'S/AGB STRATEGIC 
INDICATORS SURVEY QUESTIONNAIRE 



Financial Profile 

Reiviiue 

Tuition and fee income 
Go\-emnieni appropriaiions 
Government grants and i-oniraas 
Pmate gifis. ffM\ and coniraas 
Endowment supp^n for operations 

( payout or yield ) 
Sales and serv ices of educational 

activities 
Sales and services ofaiL\iliar\' 

enteqirises 
Sales and sen ices of hospitals 
Other sources 
Independent operations 
Total revenue 

Cummt Hxjk'iulitiiri'i hy h'laiUivii 
Instrunion (including departmental 

research ) 
Organized research 
Public senice 
Ac"adcmic support 
Student services I excluding student 

aid aw'ards I 
Instiiiiiional support 
Plant operation and maintenance 
Expenditures on auxiliaries 
Lxpendiiiires on hospitals 
Expenditures on independent 

operations 
Student financial aid 
Total exjiendiiures 
Cmreitt lixfmidiim-i by C%ec/ 
'Sages ;uid salaries (by t;pe of 

employee) 
Fringe benefits 

Interest pa\men(s to outside entities 

Rdana' Shxi. A«'/n 

Current funds 

Endowment Uxik \aliie 

Plant and cqiiipmeni 

Other isseis 

Total assets 

Hakwa' Sbeel: l.icihililics md I'und 
Hakmu-i 



Ciin'L'nt liabilities 
Short term debt to outside entities 
Long lenii deb! to outside entities 
I-'und balances 

Total liabilities and fund balances 

Physical Plant Detail 

Financial 

Beginning of ye-ar\-aluc 

Depreciation for the year 

IJetiremem of plant 

Addiuons to plant I new 

construction) 

F.nd of year value 
Plant in\eniory and condition 

Gross square feet ( by rvpc of 
facility I 

Estimated del'en-ed maintenance 
backlog (f) 

Libraries and Information 
Resources Detail 

Library holdings 

B«)k and monograph \iilunies 

Journal subscriptions 
Information resources 

Microcomputers supplied for stu 

dents' use 

Endowment Detail 

Beginning of \ear market value (by 

fiind type I 
Return on in\estmeiit (by t\pe of 

return) 

Other additions to endowment t b\ 
t\pe ) 

Subtractions from endowment 
Normal support for operations 
Special uses ( e.g., to ci ft er deficits I 
Total 

End of year market \ alue (by fund 

type I 
Students 

Fall enrollment ( head coiini & FII. 
by level) 

Fall fit: enmllnient b\ EHOG category 
and le\el 



Fall FfE enrollment bv gender and 
level 

Fall FfE enrollment by field of study 

and level 
Degrees awarded by level 
Admissions data for the flill year 

Number of applications 

Number of offers of admission 

Numlx-r of matriculants 
Geogrppliic dispersion of entering 

students by le\'el 

Number of states represented 

Student head count from home 

state 

Students from outside the L\S. 

;ind OsMih 
Tuition ;ind financial aid 
Published charges 
Financial aid head counts by npe 

of aid 

Financial aid dollars by type of 
aid 

Facult>' and Staff 

Faculty- numbers ( full and pan linie. 

by rank) 
Regular faculty FTE 

By field and rank 

By EEOC category and rank 

By gender and rank 

Percent faculty o\ er 60 years old 
Faculty gains anil losses for the year 

by rank 

Head count at the beginning of 
the year 
In hire 

\'oluntary termination 
Termination by dejth or di.sabiliiy 
Termination by the in.siitution 
Change category (e.g.. nonienured 
to tenured) 

Head count ai the end of the year 

Sponsored Research 

F-xpendiiures for organized re.search 
r S. go\'emiiieni (direct and indi 
rea.hy major ageniT) 




lABLE 3 (continued) 

MAJOR HEADINGS IN THE PETERSON'S/AGB STRATEGIC 
INDICATORS SURVEY QUESTIONNAIRE 



Suite and l(x.iil giM. agencies 
Domestic coiporjtions &• cuiixirjie 
foundations 
Other domestic private 
foundations 

Foreign governments, (.cq-'ii 
rations, foundations 
BeqiiesLs and gifts from li\ ing 
individoaK 

Other outside sponsors 
In.stitutional fiind.s 
Academic-year faculty- saian- otT\>, ^ 



Percent of regular faculty membei > 

who are prmcipal inwstigators 

nil .s|1on^(l^ed prcijeds 
Research prcipo.Siil and auard statistics 

Proposiils sent to |iotential otiiside 

sponsors 

.■\\vards received from oinside 
sp( inM .is 

Fundraising (Other than for 
Sponsored Projects) 

Dollars raised during the year by 
source 



Dollars raised during the year by 
use 

Designated or restrided for cuaent 
operations 

Designated or restricted for student 
financial aid 

Designated or restficled for 
endownient 

Designated or re.stricied for plant 
Percent of living alumni who are 
aclive donors (e.g.. hav'e given 
during the last th e years 



.\'ote: The authors plan to olfer the data on a suh.scriplioii h.Lsi.s. which would .illow in.stinition il participants 
to compare their own data, \ariahlf by xanahle. with a .st-lected pmfik- of iastitutions and elements. Like the 
NACITBC) .study, research imi\ersiiies are underrepre.sented in the cunent data set. !iut this sune\- lias more 
Indicators on faculty and .student.s than NACCBC) s pro|ect. 

Source. Ma.s,s\- and .Meyerson IWi. pp. iH i9. 

t'cKiise.s on the ability of gtadi.iating college .seniors to "think 
cTiiicali\ , cominunieate effectively, and .solve problem.s" 
(Natioiiai Center for Eckication 19941:)). A feasibility study 
developed in fall 1993 fcx-u.ses on three specific if^formational 
domains: in.stitittional reciuirements, instructional practices, 
and student behaviors (Ewell I99.3b; National Center for Edu- 
cation I99-il)). 

NCHEMS is working with a mmiber of in.stitutions to design 
sy.stems of good practice indicators (.see table 4). Approxi- 
mately 60 different items can be identified and collected 
through a variety of methods of gathering data. Such infor- 
mation can be u.sed to jjrovidc a regitlar benchmark for as.sess- 
ing collective progress in undergraduate education. Like ear- 
lier indic.-ator projects, including the Sl^ s>'.stem's effort, such 
information about an in.stitution can communicate concrete 
results easilv md succinctly to parents and legislators on what 
kinds of experiences their children and con.stituents will 
encounter at a particular college or university (see Hwell 
1994c; National Center for Education 1994b; National Center 
for Higher I99.3b). 
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TABLE* 



HEADINGS FOR THE NCHEMS INDICATORS 
OF GOOD PRACTICE PROJECT 

Institutional context and Lise of resources 

CurricLilLim class stiuctLirc and retiLiirenients 

Teaching practice and the classroom environment 

Student behavior and the campus environment 

Term registration records 

Transcript studies of graduating seniors 

End of course course evaluation sun-ey items 

Faculty surwys 

Student suneys 

Periodic course syllabus studies 

Periodic studies of the libraiy and other academic support areas 
Sduncs Kwell 1"-W31i, National Cx-mvr for Kduaition IWib. 



The ECS project 

With a 1992 grant from tiic r-und for tiic Impro\-ement of Post- 
sccondaiA- Edtication (FIPSE), the Education Commission of 
the States (ECS) is conducting a ten-state study to fiimish pub- 
lic polity makers with a set of indicators useful for monitoring 
the educational tiuality of state systems of higher education, 
and for impro\'ing teaching and learning on campus. The ten 
states included in the study— Colorado, Florida, Illinois, Ken- 
tucky, New York, South Carolina, Tennessee, Texa.s, Virginia, 
and Wi.sconsin- were selected because of their existing Indi- 
cator sy.stems or their efforts to develop an indicator .sy.stem. 
The objectives for this project link the .search for education 
indicators to concepts of what cjuality means as an aspect of 
teaching and learning for .states, systems, or campuses. With 
such a di\ erse group of representative .states, the ECS project 
attempts to .start the di.scu.ssion with a concept of quality and 
its implications for a set of performance .standards and expec- 
tations for higher education as a whole. From there, it is 
reviewing the efforts of the pilot .states and developing a con- 
ceptual framework for asse.ssing state or sy.stem performance 
as it serves the needs of various constituencies— the .state, 
the public, and students. A number of publications on per- 
formance indicators are being developed as a result of this 
project, including a report on each .state and .se\'eral related 
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[lublications on performance indicators and undergraduate 
education (see, in panicular, Ruppert 1994e). 



Patterns of Performance Indicators for the 1990s 

Tiic de\'elopmcnt of performance indicators in tlie 1990s 
differs from that in the 1980s. Tlic 1980s went through a 
number of evolutionar\' stages, hut tiic continuing f(K-us of 
polity was on quality, undergraduate education, and produc- 
ti\ir\-, among others. Several distinct new patterns have 
emerged in the 1990s, however (see, in particular, Ewell 
199'ia). 



A concern with both quality and efficiency 

The concern with quality continues to capture the attention 
of ix)lic\' makers in the 1990s, but this theme now coincides 
with states' greater interest in economic dc\eIopment and 
the emergence of higher education's second recession in the 
past ten years (Blumen.styk and Cage 1991; Minter 1992, p. 
26). A.S colleges and uni\-ersities began to reco\'er from the 
economic slump in 198'', economic decline began again in 
FY 1990, with public four year colleges and uni\'ersities expe- 
riencing the sharpest decline in growth of educational and 
general budgets. Total .state approjiriations to higher education 
in f-y 1992, in ab.solute terms, were smaller than in the preced- 
ing year, and appropriations in FY 1993 failed to keep piace 
with inflation in three-fourths of the .states (Folger and Jones 
1993, p. S). Although early projections were that this decline 
would be even deeper than the 1987 recession, the rece.ssion 
appeared to ameliorate in 1994 (Cage 1994, p. A19). At the 
same time, skepticism among .state and federal taxpayers 
about what they are getting from universities for their tax dol- 
lars has cau.sed a growing Iegislati\'e interest in efficiency and 
productivity, particularly in faculty workloads, which has 
become linked with the continuing concern about qualit\- 
("Liniversities Grapple" 1994). Tlie most immediate result 
of this pattern was a greater link between planning and budget 
allcK-ations; in lexas, for example, Hou.se Bill 2009 ( 1991 ) 
and .Senate Bill 1332 ( 1993) required state plans for all .state 
agencies, including higher education. In their requests for 
appropriations to the legislature, Texas higher education agen 
cies were retjuired to link their plan, its .stated gt)als, and the 
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related fiscal request ys a way of better connecting fiscal 
accountabilit>' with promised results. 



Quick implementation with little prior 
conceptual development 

By 1994, some 18 stales had a performance ir>dicator system 
in place, the n iajoriiy of them developed and implemented 
in the previous three years, legislation by fax prevailed, with 
imitation frequently copying pcx)r quality indicators from one 
state to another ( although some good features also were 
copied). On the positive side, this pattern suggests awareness 
of common problems and a quick response from legislators 
by using the same solutions. 

More deliberate links between planning/ 
resources and indicator systems 

The 1980s witnessed few major succe.ssful state polio,' efforts 
linking planning and budgeting, huleed, considerable cv; 
dence in the literature successfully linking planning and 
budgeting has not proved effective in practice, despite the 
.sanctity in its belief as an article of faith (Schmidtlein 1989- 
90). A notable exception is the ca.se of Tenne.s.see and its 
highly published experiment with performance funding 
(Banta 1938; Marche.se 1985). Texas also toyed with the idea 
of performance funding, but the legislature has yet to adopt 
the system (A^shworth 1994; Bateman and Elliott 1994; Gaither 
1993). Despite the "renaissance," however, the "political real- 
ity may prohibit its Rill implementation, [as] performance 
budgeting tends to have little pork barrel appeal"" (Lasher 
and Greene 1993, p. 448). 

Wliile performance funding, along with the national move- 
ment to reinvent government and transform the public sector 
(Osborne and Gaebler 1993), is witne.ssing something of a 
"renaissance," performance budgeting paradoxically had its 
beginnings in another reform period— the Progressive Era- 
in New York City during the mid- 1910s (Mike.sell 1982, p. 83). 
It again surfaced in the late 1940s during the second major 
stage of the formation of public administration and planning 
(Meisinger and Dubeck 1984, p. 185). In all three periods, 
the budget was viewed as an in.strument to control expen 
ditures, with the intent of improving efficiency. 

It is unclear whether performance Rinding will be the fi.scal 
lever to accompany the current reform movement, but .some 
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npe of incentive funding appears to be destined for connec- 
tion witii state performance indicator systems in tiie 1990s. 
A critical question is wiietiier sucii funds will be taken from 
base funding or be added as suppiementar\' funding. 

The deliberate concentration on comparisons 
of institutional performance in current 
performance indicator systems 

Tiie 1980s found legislatures and state higher education 
hoard;; reluctant to press for institutional comparisons; 
in.stead, they emphasized institutions' ro/z/^/arv improvement. 
Institutions were given the opportunirv' to make improve- 
ments voluntarily (Aper, Cuver, and Hinkle 1990), but by the 
1990s policy makers' patience was wearing thin with insti- 
tutions' perceived foot-dragging and a more forceful pattern 
of required comparisons is emerging. 

Policy makers' greater resistance 
to institutional pressure 

The 1980s found legislators more willing to compromi.se con- 
cerning propo.scd requirements for indicator s\'stems, but the 
1990s find greater resistance to iastitutional prote-sts about 
such matters. Several state educational s\'Stems— New York, 
Virginia, and Wisconsin, in particular— have engaged in 
preemptive .strikes. Perhaps sensing the inevitability of 
.stronger accountability and performance .S)',stems, these .states 
and .s>'.stems of higher education t(K)k the initiative and devel 
oped (or are dexeioping) .s\ '.stems of performance indicators 
on their own (often requiring significant statistical and data 
collection) to demoastrate to the public and to polia- makers 
their willingness to comply with requirements for account- 
ability. The University of Wi.sconsin .system, for example, is 
working with its .state government to develop better measures 
of accountability, 'I"he governor established the Accountability 
Task Force, which included university repre.sentatives, in 
March 1993 to identif\- .specific indicators to measure the uni 
versitN' .system's performance in seven areius: quality, effec 
tivene.ss, efficienc")', access, diversity, .stewardship of a.s.sets, 
and contribution to compelling .state needs (see table S). Core 
indicators were de\'eloped, and each in.stitution also devel- 
oped measures of accountability Performance will be re 
ported on a regular hasi.s. To date, the re.spon.se to the elTorl 
has been positiw, helping communicate to policy makers ;-nd 



TABLE 5 

SOME ACCOUNTABILITY INDICATORS FROM THE 

UNIVERSITY OF WISCONSIN SYSTEM'S 
ACCOUNTABILITY FOR ACHIEVEMENT PROGRAM 

Quality 

• Siudciit .satisfattinii sun cys regarding atvessibility ol (acuity 
inside and outside the elas,snx)m; quality of instnietion; aea 
deiiiic and career advising; course availa'.iilitx-; insohenient in 
faculty based research; institutit)nal responsiveness to students' 
concerns, etc. 

• Alumni satisfaction surveys conducted fwe years after graduation 
regarding their assessment of the qualit\- of education received 
with respect to career preparation, ins oK ement in cis'ic. cultural 
affairs, coninnmitx- serx ice. and cultural achies ement 

• Percentage of all undergraduate course enrollments and instruc 
tor hours taught by instructor t\pe— faculty, instructional aca 
deniic staff, and teaching a.ssistants 

Effectiveness 

• Results of systemsvide .sophomore competency tests (ACf-CA/M') 
or other Board approved alternatis e tests 

• GraduatiiMi rate of undergraduates by fi.ill and part time .status 

• GRE, LSAT. and test .scores of .students, job placement rates, pro 
fessional and graduate .school acceptance rate.s. pa.ss rates on 
licensure exams, and other data showing postgraduate expe 
rience and success 

Efficiency 

• .As erage credits to degree for .students graduating with a B.A.; 
proportion of students graduating in four s ears, by residency 

• Proportion t)f paxrolled PTE positions and expenditures 
accounted for b\- in.stitutional support, academic supptirt, in.struc 
tion. and .student serv ices 

Access 

• Percentage of ciualified undergraduate resident .students accepted 
for admi.ssitm; percentage of Wisconsin high .school .students 
who cnrt)ll immediately in the l^S 

Diversity 

• Progress towartl F.EO M goals in hiring, promotion, and tenure 
of w\)men and miiH)rit\- facultx' and statt 

• Number and rcsidenty of minority students b\ race and eth 
nicitv; graduation rates of .same 

• Incidents of .sexual liara.ssment 

Stewardship of Assets 

• Recruitment ;ind retention rates of facult\- and expenditures on 
facults- and .staff development 
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TABLE 5 (continued) 

SOME ACCOUNTABILITY INDICATORS FROM THE 

UNIVERSITY OF WISCONSIN SYSTEM'S 
ACCOUNTABILITY FOR ACHIEVEMENT PROGRAM 

• Progress on addressing preventati\'e maintenance needs ofuni 
versity facilities 

• Number and se\'erity of accidents, injuries, and identified kxsses 
or ,safety risks 

Contribution to CompeUlng State Needs 

• ResLilLs of employer ,sur\-eys regarding their a.s,se,ssnient of the 
aireer preparation of l'^X'S graduates, their .satisfaction with the 
responsivene.ss of the IWS to their needs for technical a.ssi.stance 
and other outreach ,ser\ iccs 

• EnrollmenLs in continuing education programs, including a 
demographic analysis of participants and an analysis of u.scr.s' 
perspectives on emerging needs and unmet demand 

-Vo/c. r-'or more dciail, ^ee I 'iiiwrsily of W'Iscoiimii Sv.sloni IWi 

the public alike that the university .system is being account 
able to the .state ("Strengthening" 1994, p. .^). F^Thaps the 
most successful and mo.st publicized of these preemptive 
-strikes to date, however, ha.s been SUNYs effort (see, e.g., 
Burke 1993b; Richardson 1994f, p. 65). 



The K-12 connection 

A current question of polity in developing performance indi 
cators is the tendency to connect reform in kindergarten 
through grade 12 (K-12) with reform in higher education 
(Ewell 1994a). Policy makers perceive that reform in kinder 
garten through grade 12 is working and that all public edu 
cation should therefore be integrated and connected into a 
continuum, or public accountability framework. Tims, polity- 
makers are thinking not in terms of K~ 12, but K-16. 'Hie .state 
of Kentuck>', which ha.s thoroughly reformed iLs K-12 program 
in the 1990s, is the most prominent example of this mani 
festation (Ruppert 1994a, p. 52). Tlie connection is present 
but le,ss obvious in other .states, and it appears the tiend could 
exhibit it,self more often in the 1990s. National organizations 
representing the K- 12 arena have been actively using blue 
ribbon committees to identify' and define essential content 



and standards for what studenis s'uould know and be able to 
perform, Tlie National Council of Teachers of Mathematics, 
for example, led the way when it released math .standards in 
1989, Standards in visual arts, music, theater, and dance, and 
.standards in geography were released in 199-4. and as of late 
1994 more national stantbrds in histor\', cixics. and gox'ern 
ment were being dex'eloped. Federal legislation promotes 
the use of national .standards or indicators in kindergarten 
through grade 12. The "Goals 2000: Educate America Act," 
signed into law in March 199.^. encourages .states to .set such 
.standards but does not \et tie funding to the u.se of national 
.standards. Tliese \-arious .standards prox'ide \ isible resources 
and models to draw upon xs state polity makers review their 
educational enterprises. Nowhere, it appears, arc .slate polity 
makers adopting the .standards wholesale, so the long-term 
connection could be less dramatic than some proiionenl.s had 
iioped for. 

Summary 

A caveat should be noted here concerning the number of vari 
ations among ,states within these major polity tiiemcs oi the 
1990s. The .states' hi.stor\' of public po\ky is manife.stly dit 
ferent according to indixidual .state's actions (Bogue 199.^). 
Georgia's legislature, for example, in recent years has pro 
vided significant independence to its higher education .sy.stem 
and ha.s prcviuced no major legislatix e acts concerning 
accountability, quality, t)r required performance reporting 
(Bogue. Creech, and Folger 199.^). In contrast, the ,state of 
Florida has a hi,stoi-\' of mandating academic program rex'iews. 
outcomes for general etkication (e.g., the College lrx'el Aca- 
demic Skills Te.st). select etl indices of productivity, and annual 
reports on a number of .state-level performance indicators 
( Florida .State 1992; Van de Water 199-4b, p, 27), The political 
party in power, peisonalities, the jiublic nnx)d, the quality 
of leadership, ,state economic conditions, and other factors 
can drix'e or accentuate differences in .states' responses. Hut 
ticspite the tendenty to copy indicators from one .state to 
another, many states clearly have taken a proactive ajiproach, 
empha.si/.ing one or more initiatives to promote or achieve 
quality- a.ssurance, Tlie following subsection reviews some 
of the approaches x'arious .states are emphasizing to achiex'e 
ixiblic accountability and tiuality a.ssurance. 
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Responding to the Challenge of Public Accountability 

The Gutvmment [isj rer\'kcvn urt amassing statisiics— they 
collect them, add them, raise them, tu the nth power, take 
the cube root, and prepare uwiderfid diagrams. But what 
you must nerer forget is that ever}' one of those figures 
comes in the fir.^t instance from the chowty dar [the Indian 
tillage watclnnanj, who just puts down what he damn 
plec4ses. 

— Josiah Stamp ( 1929) 

'nic 1990.S clearly are witnessing a change in policy makers' 
attitudes toward higher education, The general tone is one 
of less voluntary institutional improvement and more man 
dated public accountability, For example, of ten "front-burner 
issues" for colleges and universities. "o\ ersight and account- 
ability" was one of the first identified (AsscxMation of Go\'ern 
ing 1994), Tlic AGB predicts that the federal go\'emment will 
become "more directly iinolved in the review of institutions" 
and that "increased skepticism about accreditation" will see 
new state agencies involved in the fiscal, academic, and ad- 
ministrati\e review of institutions ( 1994), 

Accreditation agencies were once regarded as trusted gate- 
keepers of quality. Now, however, they are being viewed, par- 
tii ularh' at the federal level, with suspicion and as a series 
of old bo}' networks in which hardly anyone ever fails or is 
held fully accountable. The new state agencies predicted by 
AGB probably will ser\'e more to monitor and enforce increas- 
ing federal laws or regulations. Some standards, such a,s those 
recently developed for state postsecondary review entities 
(SPREs). are becoming de facto federal performance stan- 
dards, which will be monitored by individual .state.s. Clear 
kxsers in this shift of power are the accreditation agencies. 
.Some accreditation agencies, such as the .Southern As.S(.K-iation 
of Colleges and .Schools, the North Central As-scK'iation, and 
the Middle .States A.s.s(Kiation, are re.sponding to federal con 
cems, but most such agencies .still appear recalcitrant. Thus, 
a theme of more centralized authority to bring about more 
public accountability- and better management, particularly at 
the .state level, will likely undedie the ways that many of 
higher education's future funding needs will be addressed. 

Partially as a result of this climate, current funding mech 
anisms for higher education mast change to fcx-us on edu- 
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cational goals rather than t ontinuir.g to promote mostly eqult>' 
in funding and access as they have for the last 20 years (Al- 
bright 1994). Higher education must now "move to a value- 
driven s7steni [that] fcK-uses on student and institutional per- 
formance and improvements" (pp. 7. 8). 

'Iliis emphasis on "a clean slate" for fiinding in the 1990s 
is another example of the way some cun-ent policy makers 
are actively seeking to provide more public accountability 
and institutional impro\'ement. In the 19H0s, colleges and uni- 
versities were encouraged to develop ro/(/^i/«n' improve- 
ments, panicularly in undergraduate education, and dcKument 
what results were achieved (Aper, Cuver. and Hinkle 1990). 
Now, however, public accountability, using fiscal policy and 
state legislation, has become a central purpose of states' 
efforts rather than a voluntary by-product, 

Other new approaches to funding would promote greater 
improvement in perftirmancc and greater public accounta- 
bility, thereby encouraging the use of performance indicators. 
The Texas Commissioner of Higher Education, for example, 
believing that such ,s7stems will be used more frequently in 
the fiiture. has developed a performance indicator .system for 
po.ssible use in Texas (.see table 6) (Ashworth 1994). A fresh 
approach to state fi.scal policy as applied to in.stitutions of 
higher education is not ncce.s.sarily designed to generate more 
flmds than other fiscal policies, for decreased flmding for 
higher education is not a temporars- condition (Folger and 
Jones 1993). Instead, the proposed .sy.stem is designed to 
ensure that states" interests and educational priorities are 
closely connected, almost .symbiotically, during the alkx-ation 
prcK-ess. This three part state budget system would include 
a base or core lump-sum budget, a capital budget, and a 
.special-purjiose budget (Folger and Jones 1993): 

• The core lump sum budget. Subject to an in.stitutioiVs 
accountability and assessment of its achieving its goals, 
the institution would be authorized to manage this 
budget, with a minimum of interference. 

• The capital budget. Tliis »-aidget would cover the acc)ui- 
sition of assets and neW' buildings. 

• The spccialpurfwse budget. This budget, at 5 to 10 percent 
of the base budget, would be tied to the state's objectives; 

• ba.sed on .some form of a.sscssment to measure progre.ss. 
such as performance indicators, the monies would be 
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used to advance ilic state's agenda througli its higlicr edu- 
cation system. ( It is important that tiiis budget be in addi 
tion to the iiasc !-)udget, not a portion of it. Otherwise, 
it is uiinerabie to elimination at the first sign of a 
decreased budget [Hollander 1991].) 

'Hie special puq-jose budget could assume several different 
forms during allcK'aiions: 
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• Block ^^rants. I'erhaps the best example of such an effort 
is Ohio's Selective Excellence Program, which existed 
from NH,^ to 1991. One part of this six-part program was 
the "academic challenge" bkx'k grant, which awarded 
$H6 million from 19H5 to 1991. Institutions used the block 
grants, which amounted to 1 percent of the .state's appro 
priaiions, to strengthen their .strong programs and develop 
additional centers of excellence (Folger and Jones 1993, 
p. 36; National Center for Higher 1992). 

• luitidliiv (ir prospL'Cliri' ( rompetitive ) JiiHciin!;. Virginia 
ha.s provided comi:)etitive grants since 19H0, and New jer 
se\-, using a ba.se plus budget enhancement model from 
19K-1 to 1990, had a dozen such programs and awarded 
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over million (although it eliminated all a)m[K'titive 
programs in 1990 91. primarily as a result of budget cuts 
and institutional opposition). .State objecli%'es, such as 
increasing the retention and graduation rates of minor 
ities, are normally funded under such competiti\'e grants, 
and institutions can use the funds in .some entrepreneurial 
way of their choosing to achie\'e the .state's objectives. 
Because such .siate objectives ha\e lower priorities with 
in.stitutions than their base budgets and core ob|ecti\'es, 
when funds arc decreased, in.stitutions seek to present 
the ha.sics firet (Hollander 1991; Morgan 1992, p. 290). 

• IitcL'iithv fiiiidiii^. Such grants are awarded on the ba.sis 
of progress toward a .state goal. Typically, a .state de\'clops 
performance indicators of undergraduate cjuality and 
cffcctiwne.ss and pro\ules "incentive funding" for an 
in.stitution that achieves or exceeds .some designated 
.standard. Colorado, for example, pa.ssed an incenti\-e 
financing law in 1993 to .strengthen undergraduate edu 
cation a.nd to increa.se hi.storically bypa.s.sed .student pop 
Illations ( Lively 199-4). Tenne.s.see's performance funding 
(instructional improvement) project, which began in 
19"'9, allowed each in.stitution to earn up to 2 percent 

of its .stale appropriations. More complex rules were 
added to Tenne.ssee's program in the 1980s, primarily to 
ensure fairness and reduce game playing (Folger and 
Jones 1993, p. r\ Morgan 1992, p. 292). 

• Stiuk'nt funding (scholarships and uvrh sttidy grants). 
A commonly used practice to enhance and enrich under 
graduate education, financial aid and .aate .scholarships 
are used to attract and help retain .star ( high-achieving) 
students. /Vnother approach is to u.se them to attract a dif 
ferent mix of .students. Texas, for example, through .special 
funding provides .scholarships to white .students to attend 
a hi.storically black college in the .state, in a related vein, 
Mis.souri pro\ides a designated amount of funding to an 
in.stitution for each minority graduate under its "Funding 
for Results" program. 

Such proposals suggest that traditional funding -A-.stems in 
higher education could contain .serious disincentives that dis 
courage a faculty member's commitment to teaching or meet 
ing state goals. Clearly, the internal reward .sy.stem .seems at 
odds with .state goals on imjiro\'ing and enriching undergrad 
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uate education. On university campuses, for example, the 
primary- and co\etcd rewartls for faculty are more merit pay, 
promotion, tenure, teacliing graduate students, researcli, and 
publication— not necessarily in that c)rder (see, in particular. 
Diamond and Adam 1993). At the .state level, .some commonly 
.stated giwls for higher education over the last decade include 
better and more innovaii\'e undergraduate instruction invoK' 
ing .senior faculty, more public accountability ("and, thu.s, often 
more control over the campus), aciiievement of collective, 
particularly economic, goals for the state, better health care 
ser\-ices, and greater access for historically undcrserved pop 
ulations, who often require remedial or developmental edu- 
cation. Clearly, the conflicting general objectives of both par- 
ties cany within them .some friction and dissonance. Tlie .state 
Vi-ields the budget ax and the legislature to .stimulate faculty, 
but the faculty can employ pa.ssive resistance. 'Ilie u.se of llscal 
polity and incentives as outlined earlier with \'aiid and reli- 
able performance indicators to measure and reward progress 
could fo.ster new vision and innovation in college teaching. 
A new approach using fiscal polity could provide the .stimulus 
needed to create an increasingly homogenous .set of goals 
for both institutions and .scxMCty. F'aculty have yet to respt)nd 
with commitment to such externally imposed accountability 
.systems (Burke 1994, p. 3). 

Although performance indicators for higher education 
became established in F,urope in the 19HOs, these activities 
are .still relatively new in the United States. Before such indi- 
cators can be effecti\'ely employed to ensure better public 
accountability, it will be necessar\- fir.st for the .states, in con 
junction with the education community, to adopt clear goals 
or expectations for their higher education .systems. If a higher 
education .system's goals are not known and are not clear, 
then how can a set of performance indicators be adequately 
applied to measure, evaluate, and reward progre.ss toward 
desired goals? 

nie ongoing ten .state prt)ject by the Etlucation Commission 
of the States has identified certain general areas around which 
.state goals and expectations are developing: effectiveness, 
efficient'>"pr(xluctivity, access, quality of undergraduate edu- 
cation, equity -tliversity, state economic development neeils, 
and contributions to reform of elementary and .secondary- edu 
cation (Ruppert 1994e). The fact that those ten states have 
focused on common topics for the improvement of perfor 



mance is further evidence that a core set of common prob- 
lems probably exists across the states. A state must delineate 
clear, measurable goals before a performance indicator system 
can be used to maximum benefit. 

Performance indicators, used now in some 18 states, with 
some exceptions contiiin 15 to 20 distinct items for collecting 
data (Ewell 1994b, p. 16). Tlie following list includes the com- 
mon core of performance indicators identified by the ten 
states in the ECS project-. 

• Enrollment graduation data by gender, ethnicity, and 
proi?ram 

• Degree completions and time to degree 

• Persistence retention rates by grade, ethnicity, and 
program 

• Number percent of accredited eligible programs 

• Remediation activities and indicators of their effectiveness 

• Transfer rates to and from tv.'o- and four-year in.stitutions 

• Pass rates on professional exams (e.g., CPA, nursing, 
engineering) 

• Job placement data on graduates and graduates" satisfac 
tion with their jobs 

• Faculty workload and producti\ity in the form of .student 
faculty ratios and in.structional contact hours. 

Other list.s. however (.see. e.g.. Taylor. Meyerson. and Ma.s,sy 
1993), include entirely different indicators (.see table ■^). While 
the indicators shov,Ti on table ';' are "critical success factors" 
for an in.stitution, the differences in li.sts further signifv' the 
conflict between ,state and in.stitutional definitions of ".suc- 
cess" (see. in particular, Cave, Hanney, and Kog;in 1991. p. 
153). Further, it is difficult, at the .state le\ el, to use .state fiscal 
policy and performance indicators to measure the achieve- 
ment of any agreed upon .state educational goals, although 
such differences can be re.soK'ed (Richard.son 1994a. p. 131). 

^Xliile the.se ECS core indicators are conmionly u.sed. several 
.states have added embellishments that di.stinguish them from 
the norm. South Carolina, for example, collects information 
on upper-division .students" in\'olvement in spon.sored re 
search. \'irginia collects information on undergraduate stu 
dents' integrati\ e experiences. A number of .states have indi 
cators regarding small classes (Wi.sconsin and Virginia) or 
cla.ss size by level (New York) in line with the commonly held 
belief that small classes have a high positive value for irnpro\- 
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ing the quality of undergraduate education— even though 
the idea is not necessarily supported by research (Williams 
et al. 1985). Such indicators based on class size could repre- 
sent another area where the validity' of some existing state 
performance measures can be called into question. 

The con' list of performance indicators in the United States 
is comparable to indicators used in other countries (see 
Borden and Banta 1994; Bottrill and Borden 1994, p. 10''; 
Cave, Hanney, and Kogan 1991, p. 172; Cuenin 1987, p. 135: 

TABLE 7 
TOP TEN "CORE" INDICATORS 

1. Overall Revenue Stmcture 

2. Overall Expenditure Structure 

3- Exce.s.s (Deficit ) or Current Fund Revenues ()\er (or under) Cur 
rent Fund Expenditures 

4. Percent of Freshman A|iplicam.s Accepted and l^ercent of Accepted 
Freshman Who Matriculate 

5. Ratio of Full Time Equi\'alent .Students to Full-Time Equivalent 
FacultN- 

6. In.stitutional Grant Aid xs a Percent of Tuition and Fee Income 

7. Tenure Status of Full Time Equivalent FacultN- 

8. Percent of Total Full Time Equivalent Employees '^lio Ai e Faculty 

9. Maintenance Backlog as a Percent of Total Replacement Value 
of Plant 

10. Percent of Living Alumni Wlio Have Gi\'en at Any Time during 
the Past Five Years 

Source: Taylor. Meyerson, and .\la.s,s\- 199.1. pp. x\--x-\l 

Sizer, Spec, and Bormans 1992, p. 151), although in some 
cases the tenninology differs. The British, for example, refer 
to attrition and persistence rates as "wastage" iind first job 
placement as "first destination" (Cave, Hanney, and Kogan 
1991, p. 95). On the other hand, citation indices (e.g., number 
of publications cited in works) and research produced are 
common international performance indicators that are used 
infrequently in the United States (see, e.g., Cave, Hanney, and 
Kogan 1991, p. 114; Hattic 1990), Instead, for example, Colo- 
rad(Vs House Bill 1 187, in effect from 1985 to 1991, mandating 
accountability in Colorado in higher education, and Colo- 
rado's 1993 "incentive financing" law emphasize .strengthen 
ing imdergraduate education and reaching students who 



would not normally go to college (Frantz 1992; Lively 1994; 
Van de Water 1994a). Indeed, a distinct difference between 
European and U.S. performance indicators is that the ILS. ver 
sion does not adequately recognize or reward graduate or 
professional training, or research. Further, the ftxrus on eco 
nomic development and job placement data about graduates 
in some of the core U.S. indicator systems makes land grant 
and professional institutions better able to benefit or comply 
with such indicators than liberal arts in.stitutions or predom 
inantly graduate and research institutions. 



Summary 

Higher education in the United States in the 1990s h;is wit 
nessed the emergence or reshaping of several trends. First, 
policy makers" mood has changed since the 1980s, when legis- 
latures seemed more willing to compromise with institutions, 
to a harder line, often manifesting itself now through the 
legislative prtx-e.ss. This trend was fueled by a decided shift 
in the 1990s to more demands for greater public accounta- 
bility' and accompanied by a shift in sentiment that saw 18 
.states" rather rapid adoption of performance indicators, mo.st 
during the first three years of the decade. These adaptations 
were often accomplished through imitation— with all of the 
flaws and virtues transmitted from one state to another. 

Accompanying these trends were responses by se\'eral pro- 
fessional organizations or institutes— NACUBO, NCHEMS, 
AGB, and others- to develop indicators. Many of these efforts 
are designed to identify- and define a set of performance indi 
cators and de\ elop normative data that will allow for com- 
parisons among institutions. 

Tlie .states involved in designing performance indicators 
share common problems (and goals) with their higher edu 
cation systems. A ' -ore set of common performance indicators 
began to evolve around these problems, cutting acro.ss a 
number of states ;ind showing similarities with international 
efforts. A major dift'erence. however, is that I'.S. indicators tend 
to fcK-us on undergraduate education or economic develop 
ment more than their overseas counterjiarts. 

Se\'eral new approaches to budgeting ha\-e been proposed 
that support the growing trend in the United States toward 
greater public accountability. Incentives to support or buttress 
.state formula funding were developed. FcK-using more on pro 
cess, outputs, and the ciiiality of the efforts, such outcomes- 
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oriented efforts were designed to provide incentives to ac- 
complish desired gcwls. 

Accompanying these efforts are questions, still unanswered, 
about the use of group incentives to bring about institutional 
change. Reform must begin with the individual — more spe- 
cifically in higher education, with the facult>''s support— for 
it to be effective. And how much is money, when dispensed 
to an institution rather than an individual, a motivator to bring 
about needed reform? A certain tension between the campus 
and the state (or Washington, D.C) appears on the horizon, 
particularly as greater centralization of decision making at the 
state level grows imminent. Tlie goals of the faculty- and the 
state legislature often appear at odds, almost guaranteeing 
some level of ongoing tension and conflict between these 
groups. Ilie regional and national role of accrediting a.sso- 
ciations appears to be eroding, being replaced by, among 
others, the federal go\'ernment in areas like financial aid and 
national performance goals. And the trend appears to be grow- 
ing, with more centralization by the states following the 
"Republican revolution" during the 1994 elections. 

Tlie use of performance indicators does not stop at our 
shores. Indeed, Americans have gained a great deal of expe- 
rience with the problems and i.ssues that fueled these reforms 
elsewhere — particularly in Europe and Australia, the fcx'us 
of the next section. 



THE INTERNATIONAL USE OF PERFORMANCE INDICATORS 



An examination of the use of performance indicators inter- 
nationally reveals a wide range of philosophies and applica- 
tions. Despite this diversity', the use of indicators generally 
occurs within a framework of increased government account- 
ability and fiscal constraint, as governments' concept of their 
role in higher education shifted from a type of "public utilit^•■■ 
to a "strategic investment" (Ewell 1991). Effective perfor- 
mance has become an integral part of the di.scu.ssion overseas 
about excellence and quality, as it has in the United States. 
Consequently, international approaches to assessment and 
evaluation share a common connection to budgeting pro 
cesses for .state systems and individual institutions. 

Background 

At no time in its history has higher education been faced with 
greater demands to demonstrate its worth ;ind account for 
its use of public resources (Kogan 1989). But a shift from pro 
fessional self-evaluation to external and managerial assess- 
ment reveals a number of common problems: 

1. The application of evaluation techniques used in industry, 
which are of limited value to higher education, especially 
those emphasizing outcomes or product rather than 
process. 

2. The change in assessment criteria from tho.se stressing 
excellence to criteria related to economic or serial 
outcomes. 

3. Questions of pcAver regarding mechanisms, motivations, 
and methcxiologies for as.sessment. 

Despite the considerable interest generated by discussions 
about performance indicators, indicators cstensibly plav a rel 
atively minor role in the actual policies and decisions of most 
governments and institutions. A number of factors limit the 
impact of performance indicators: 

1. Tlie connection between indicators and .selecti\-c funding. 

viewed as threatening to in.stitutions. 
2 Issues of validity and reliability in the selection and appli 

cation of performance indicators. 

A lack of conceptual agreement ben\'ecn governments 

and in.stitutions on the use of indicators (Si/.er, S|)cc. and 

Bormans 1992). 
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Nevertheless, consensus is growing in Europe regarding the 
usefulness of performance indicators in reshaping educa- 
tion—particularly when they are complemented with more 
cjualitative information Qongbloed and Westerheijden 1994). 

This section summarizes the literature on the use of per- 
formance indicators in "selected countries outside the United 
States, compares indicator systems in u.se around the wodd, 
and summarizes the major themes presented in the profiles 
of systems used in other ccjuntries. 

The Use of Performance Indicators 

In light of the wide range of assumptions, approaches, inten 
tions, and levels of sophistication present in performance indi- 
cators, some criteria are necessar\- to compare them so that 
their usefulness and replicability in other settings can he 
assessed. The following comparative criteria arc used as the 
framework to describe performance indicator ,s)',stems in Can 
ada, Australia, and five countries in Europe and to show the 
links among organizational culture, institutional change, 
governmental and managerial priorities, and pcdagog\': 

1. k)cus of control 

2. Degree of government invoK'emcnt 

3. Fwus of performance indicators 

4. Sources of variation in quality 

5. Data selection 

6. Intended audiences 

7. Emphasis of use 

8. Impact on students" learning 

9. Relationship to institutional mi.ssion (Nedwek and Neal 
1994b). 

These nine dimensions provide a framework for the exam 
ination of similarities and differences in the development of 
indicator .s^'stems in the selected countries, as well as a iheo 
retical basis for the analysis of .strengths and weakne.s,ses in 
each initiative. 

Locus of control 

Change within higher education in.stitutions results primarily 
from one of r?\'o sources: internal forces of reform or external 
demands from the environment. Tlie organization is engaged 
in a .struggle o\'cr control with its environment ( Peterson and 
.Spencer 1993), but because of higher education's dual .system 



UU 



of governance (administrators and professionals), the use- 
fulness of traditional management techniques, such as stra- 
tegic ■'control" mechanisms, is limited (Dejager 1992). Never- 
theless, the mechanisms used for control in organizations 
including higher education can be classified into clans, 
bureaucracies, and markets (Ouchi 1989). Clans seek to con- 
trol members' beha\-ior through shared values, traditions, and 
social structures, such as tho.se found among iaculty in depart- 
ments, schools, or universities. Bureaucracies control through 
hierarchical authority and rules, such as state agencies or 
accrediting bodies. Markets rely on forces like competition 
and pricing to shape organizational behavior. 

Degree of government involvement 

■Wliile goN crnment involvement can be viewed as a subset 
of l(x:us of control, it is di.stinct in this discussion because of 
the interplay of perceptions between go\ emment officials 
and academic leaders (Nedwek and Neal 1994b). The level 
of government involvement is dictiited by the scarcity of 
resources and the attitudes, beliefs, and values within the po- 
litical culture (Boyer 1990; Ewell 1989). These perceptions 
in turn shape the form and substance of a.ssessment s\',stems. 
GoN'ernment involvement ranges from direct to indirect to 
lai.ssez faire. Direct in\'ol\'ement results from state-mandated 
.systems with prescribed mcthcxlologies for the design and 
use of performance indicators. Indirect control encourages 
limited delegation of authority to in.stitutions to accommodate 
diversity within the context of broad goal .statements. A 
lai.ssez-faire position exi.sts when government leaders show 
disinterest in educational policy or when coordinated efforts 
are largely voluntary. 

Focus of performance indicators 

Performance indicators should be directly related to the in.sti- 
tution's functions and goals (Ball and Wilkinson 1992; Dcx-hy 
and Segers 1990). Tlie foci of indicators include effecti\-ene.ss 
( how well the institution accomplishes its stated goals and 
objectives), increasing efficiency (what it costs to achieve 
those goals), or improving the institution's economic envi 
ronment (how to save money yet achieve the goals). 

Sources of variation in qtmlity 

Hie philosophy of continuous and .systematic improvement 
.stresses the importance of reducing \ariation in production 
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design and processes (Deming 1986). Special causes of vari- 
ation (such as individual performance) must be kept anaKt- 
ically distinct from common causes (those attributable to poor 
institutional prcxesses), for common causes (variations in 
design, materials, technologies, and super\lsion) will sub- 
stantially affect the quality of an institution (Dill 1992). Special 
causes of variations are exceptions to the normal prtK'ess that 
require quick detection and administrative action to eliminate 
(Chaffee and Sherr 1992). Analyses of common causes of vari- 
ation often result in an overemphasis on the connection be- 
tween inputs and otitcomes, resulting in a call for increased 
assessment of the prcK'css as a component of institutional eval- 
uation (Nedwek 1994; Schilling and Schilling 1993). 




Data selection 

In the absence of consensus on conceptual frameworks, the 
options for gathering and selecting data vary widely but usu- 
ally reflect a classic mechanistic view of organizations. While 
some .systems select data about input, processes, and out- 
comes to assess quality (Ball and Halwachi 1987; Chaffee and 
Sherr 1992; Scottish Higher Education 1992), most institutions 
still concentrate primarily on measures of input (Johnes and 
Taylor 1990). Methods of collecting data reflect a quantitative 
bias and tend to include such mea.sures because the\' are read- 
ily a\'ailabie and easily quantifiable (Ball and Wilkinson 1992; 
Elton 1987). TlTis quantitative bias is revealed in the devel- 
opment of pi-ox\' measures of prcx-ess, such as human re- 
st)urce ratios (faculty- 'staff and faculty- per FTE student, for 
example) (Schmitz 1993). In an era of fiscal constraint, finan- 
cial ratio analysis has emerged as a way to assess financial per- 
formance, institutional priorities, and comparisons of peer 
organizations (Chaix)tar 1989; DiSalvo 1989; KPMG Peat Mar- 
wick 1990). 



Intended audiences 

Higher education institutions produce performance indicators 
for a variety of audiences, including governing bodies, legis- 
latures, students, and faculty. Tlie following t)polog\' catego- 
rizes these audiences by relationship to the institution and 
anticipated outcome of the information: 

1. Internal decision makers (administration faculty) 

2. Internal policy makers (board administration) 

3. External decision makers (parents students) 
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4. External policy makers (government agencies accredi 
tation bodies) (Nedwek and Neal 1994b). 

Emphasis of use 

Performance indicators ha\ e four primary' purposes: (1) moni- 
toring general conditions and contexts; (2) identifying prog- 
ress toward specified goals; (3) illuminating or foreshadowing 
problems; and (4) diagnosing potential sources of identified 
problems (Dading-Hammond 1992). Tliey can serve two addi- 
tional purposes as well: the alkxation of incentive funding 
and resources (Banta 1986; "Development of 1985); and po 
litical symbolism (Ewell 1989; Kalsbeek 1991; l^viton and 
Hughes 1981; Nedwek and Neal 1994a; Weiss 1977). 

Impact on students' learning 

Most performance indicator systems reflect primarily rcspon 
siveness to external pre.ssures and demands. As a result, little 
attention is given to prcx'e.sses within the organization, espe- 
cially the learning environment (Nedwek and Neal 1994b). 
Sen.sitivity to pr(x:e.ss or the program's effects on .students' 
learning can be direct, indirect, or assumed. Direct impact 
exists when the model intentionally yields data to validate 
students' achievement as related to the iiLstitution's mi.ssion. 
Indirect impact represents models that fo.ster a concern about 
variations in outcomes and links to the delivery of programs. 
Assumed impact, the mo.st common model, exists when .stu 
dent outcomes sen-e as a leap of faith from the program out- 
line—for example, the connection between improvements 
in productivity and enhancement of academic quality ( Massy 
and Wilger 1992). 

Relationship to institutional mission 

Performance indicators vary in the degree to which they arc 
related to programmatic, in.stitutional, or sy.stcmwidc mi.ssions. 
WhWc mi.ssion .statements .should provide a starting point in 
the development of indicators, the actual relationship might 
be direct, indirect, or a.ssumed (Nedwek and Neal 1994b). 
A direct relationship exi.sts when the mission .statement clearly 
promotes the instit.ution's distinctiveness, and such a .state- 
ment .should therefore include clear goals about intended 
outcomes, clientele to be serv ed, and .steps to ensure achieve 
ment of the mission (Bogue and Saunders 1992; Newsom and 
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Hayes 1990). An indirect reiationsiiip exists wIilmi the mission 
statement and the perfonnance assessment system suggest 
less precise standards. Assumed relationships cx'cur when mis- 
sion statements and perfonnance indicators are independent. 

Case Studies 
Great Britain 

Background. Puipid increases in enrollment, the demand 
for the efficient use of resources, and the final abolition in 
1992 of the binan.' educational .sy.stem (two di.stinct .sectors — 
pt)Ktechnic .schools and colleges, and universities) ha\'e set 
the tone for significant changes in gcn ernance, funding, and 
allocations in Great Britain. Public in.stitutions in Great Britain 
include ISO former poKiechnic schc/Ols and colleges and 43 
universities (excluding the Open University). Although the 
poKtechnic .schools .still maintain a more \-cK-atit)nal focus, 
.st)me of the larger in.stitutions have an academic reputation 
higher than the universities (Goedegebuure, iMaa.s.sen, and 
We.sterheijden 1990a). 

The universities enrolled more than 3^2.000 full-time .stu- 
dents and nearly 52,000 part-time students at the outset of 
the decade (AsscK'iation of Commonwealth 1993). In recent 
years, in.stitutions in Great Britain experienced rapid grovslh 
in .student enrollments .similar to that of the United States dur- 
ing the 1960.S. Full-time enrollments al.so are increasing .stead- 
ily, with an 11 percent gain benveen 1988 89 and 1990 91. 
Government projections of enrollment call lor a 33 percent 
increase in demand by 2000 (Kells 1993). 

Unlike their coiinterjiarts in the United States. Briti.sh in.sti 
tution.s tend io be quite small. a\'eraging, as late as 1988, 
around 6,000 students (Johnes and Taylor 1990). At the outset 
of the decade, provisional expenditures for further and higher 
education in the U.K. reached £6.8 billion ($1 1 billion) 
( WhitakKTS Almamtc 199h ). 

Environmental context. Great Britain's higher education 
.system of universities and public in.stitutions has experienced 
a revolution of rising expectations during the past fe\\' 
tiecades. Tlie introduction of performance indicators to the 
lexicon of higher education management in Great Britain 
marked a major shift in power that welcomed a consei"\'ative 
ideology to go\'emment in 19"'9. Gcnernment's perspectiw 



on hi}»hcr education became one of "value for money," 
cmph;isizing economy, efficiency, and effectiveness (Goe- 
degebuure, Ma;issen, and Westerheijden 1990a). 

With rising inflation and a sluggish economy in the 1970s, 
the British government no longer could tolerate its system 
of bkvk grant funding by the University Grants Committee 
(LIGC). Cuts in government support in the early 1980s sig- 
naled a major shift in the way universities and colleges would 
be funded (Williams 1990), ;ind the dramatic reduction in 
government funding and the elimination of block grants were 
strongly criticized %^^thin the academy ( Kogan and Kog;in 
1983). A second major factor in the evolution of performance 
indicators wius the publication of a government "green paper" 
( "Development of 1985) that spoke to the use of indicators 
in the aikx-ation of resources (Ball and Wilkinson 1992). 

Suggestions tor major organizational changes to higher edu- 
cation in a government "white paper," Higher Education: A 
New Frcwu'uvrk {Dcpi. of Education and Science 1991), were 
incorporated in the Further ;ind Higher Education Act of 1992 
and the Further and Higher Education (Scotland) Act of 1992. 
Tliese acts eliminated the binar>' system of higher education 
by removing the distinction between services provided by 
universities (funded in England and Wales through the Uni- 
versities Funding Council) and by polytechnics and colleges 
(funded through the Polytechnics and Colleges Funding 
Council). All institutions in England, Scotland, and Wales are 
now funded through three Higher Education Funding Coun- 
cils, which were empowered to obtain extensive institution- 
al data in support of their allegation of funds (Green 1994, 
p. 4). 

The new funding approach is sensitive to institutional mis- 
sions. 'Hie former polytechnics, historically vcKational in 
orientation, were to be judged against teaching performance 
as well as success in applied re.search. The older universities 
were expectcxi to be assessed using a "research assessment 
exercise" that produces quality ratings of the universities' suc- 
cess in research (Universities Funding 1992). A fivefold rating 
scale of productivity- in research was initiated in 1993 (Kclls 
1993), with criteria emphasizing international reputation as 
the key to the highest ratings. Tlie methodology the councils 
use in funding research is ba.sed on quality of the research, 
the contract, and development. The research assessment exer 
ci,sc in 1992 produced a four-volume set of measures for use 
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by tlie councils (Jongbloed and Westerlieijdcn 1994 ). 

Tlie relationship between higher education ;ind government 
continues to change, with a lojsening of tics between insti- 
tutions and local government (Brennan 1990). Despite these 
changes, the fear persists that the development of national 
indicators will incn-itably "nationalix.e institutional polity mak- 
ing" (Ball and Wilkinson 1992, p. 11). Tlu' Further and Higher 
Education Act of 1992 fuels some of these fears. The capacity 
to gather and analyze data, for example, was greatly impro\-ed 
with the creation of a new central agency, the Higher Edu 
cation Stati.stics Agency (HESA). which is charged with data 
collection in support of decisions by the funding council.s. 

Use of performance indicators. Th.e British model blends 
external indicators with peer judgment to maintain quality 
assurance in higher education. Di.stinctiv)n.s are drawn among 
quality audit, qualiry control, and quality a,s.se.ssment; respon 
sibility for control re.sts with in.stitutions, for audits with the 
Higher Education Quality Council, and for as.scssment with 
the new funding councils (Gordon 1992 ). Tlnu.s, a dual .sy.steni 
of control exists: one administrative and the otber professional 
(Dejager 1992). 

In the past 25 years, normati\'e mechanisms (a "dan" form 
of control by the academy) ha\ e been used as means of 
organizational control (Dill 1992; Ouchi 1989). The intro- 
duction of centrali7.ed indicators places additional pre.s.sure 
on in.stitutional autonomy (Frackmann 198^-, Kogan and 
Kogan 1983). ,Some critics argue that, for the former poly- 
technics, performance indicators repre.sent tbe lossofaca 
demic control and the expansion of managerial control 
(McElwee 1992). 

An important force influencing the allocation of resources 
for higher education in Great Britain was tbe concept of "com 
petiti\'e tendering" as a funding mechanism (McX'icar 1990). 
In this approach, the stale no longer was to provide a ser\ icc; 
instead, it purchases .student places that are subject to bid by 
.service providers. England, Scotland, and Wales each ap 
pointed representatives to a funding council that acts as a pur 
chasing agent of services based upon a complex alkx-ation 
formula (Higher Education Funding 1993; Temple and Viliitc 
church 1994). .Some argue, howe\'er, that increa.sed reliance 
on .student fees has eroded the power of tbe funiling councils 
(McVicar 1990). Each university in Great Briuiin appears to 



hf hcconiiiij^ more cnti'C'prcni'uri;il as iiKirkct loa cs ot suppK' 
and doniand replace national planning CUniplr and 
NXIiiieehurdi 1994). 

Multiple meanings and t'oei ilot (lie lanilscape of perlor 
iiianee indicators in Great Hritain. The C.oniniiitee of \ "ice 
Chancellors and I^rincipal.s (Q'C.P) suggested three c ategories 
of c|uantitaiive measures: iniernal. external, ami o|;eraiional 
(tkill and Wilkin.son Wl: Committee 19H6). l-ach measure 
holds niiiliiple meanings that often fail to |-)rovide "back 
ground inlbrmation needed to interpret the data" ( I iarris and 
I)(K-hy 1990). 

Political culture and context appear to .shajx' the focus of 
indicators ( Si/.er 1992 ). and the process of inlrodiicing per 
formance inilicators influenced the suh.stance of the indicators 
them.selves. They are neither value neutral nor process inde 
pendent (l-indlay 1990; Goedegehimre. Maa.s.sen. anil \Xes 
terheijden 1990a). Critics have argued that most indicators 
reflect the ea.se of gathering data more than a conceptual 
framework (lieNX'eert 1990; Idton I9K7), rather than being 
developed through ideiitif\'ing the puiposes to he ser\'ed iiy 
the inciicators. which in turn woukl .shape the bro;ider con 
ceptual framework (Yorke 199 Ih). 

The attitudes, helief's, and v;iliies of key .st;ikeliolders con 
ccrningrlie role of education in .soe'iety ;ind the relation.ship 
herween j>()veminent and the educational eomniunity should 
determine the design of inilicators. '"Hie more the objectives 
behind higher education j-joliiT can be regarded as an expres 
sion of the more general political culture of a country, the 
more the policy - -including any objectives |that| are jxirt of 
it - will be accepted" ( Si/.er. Spec, and Bormans 1992. pp. 
138 In the Hriti.sh public .sector, government inspectors, 
rather than peer.s. are ii.sed to judge (juality ( Cioedegehuure. 
Maa.ssen, and Westerhcijden 1990a). Ciradually. the role of 
higher education h;is come to he viewed ;is :in agent or force 
that should contribute to the national economy (.johnes ;inil 
Taylor 1990). 

Mo.st indicators in the British initi:itive are expressions of 
a meehani.stic view of education (Hltoii I9KS). A.s ;i resiili. 
most can be grouped into measures of input, process. ;md 
output ( Findlay 1990; jobnes and 'lay lot 199(1). by 1992. the 
number of Indicators bad expanded to dS; although much 
of their emphasis is on ininii a..d proxy measures ol pn x ess. 
mea.sures of outcome, such as graduation and coniideiion 



rates, are related directiy to unit coses. Thus, cost per student 
is a common criterion consistent with tiie C\'CP's management 
stati.stics m(^del of performance indicators. The use of regres- 
sion residuals provides a more sophisticated statistical ap- 
[iroach to determine expected costs per student compared 
with actual costs Qohnes and Taylor 1990). Lxplanator\' var- 
iables include ■"subject mix" to help explain variations in ncm- 
completion rates, institutional factors (faculty/student ratios 
and length of course .study, for example), and the like. Similar 
to efforts in the United States, the s\'Stematic and comparative 
use of indicators should account for variations in .students' 
abilities and the environmental conteict. Mo.st critics argue 
that indicators should, but .seldom do.'flow from in.stitutional 
mi.ssion (Cuenin 198^-, Dcxhy and .Segers 1990; Findlay 1990; 
.Segers, Wijnen, and D(K"hy 1990), but while tailoring indi- 
cators to in.stitutional mi.ssion is important, designers of indi- 
cator .systems niu.st be carefjl to minimize the development 
of performance indicators a.s "reductionist mechanisms " 
(KelLs 1992a). 

In the United Kingdom, performance indicators in general 
use include: 

1. Number of applications per undergraduate place 

2. [Undergraduates" entr\- scores 

3. Cla,ssification of or number of honors degrees awarded 

4. Wasuige rates of students 

5. Staff workload 

6. Employability of graduates 

"'. Postgraduate completion rates 

8. Research grants and contracts 

9. Re.search .studentships awarded by prominent bodies, such 
as research councils 

10. Records of publications 

1 1. .Students' loads 

12. .Staff/ .student ratios 
1 .1 Unit co.sts 

\i. Staff size by type 

1^. in.stitutional revenues and expenditures 
16. Departmental ecjuipment and recurrent grants and ex[K'n 
diaires (Cuenin 1987; Kells 1990). 

NXliile the major fcx'us has been on input variables, .some 
effort is under wa\' to assess productivity in research Much 
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of the work still uses traditional measures, such as peer 
reviews, publications, citation indices, and research revenues. 
Variations in research output are explained in part by process 
factors, such as ratios of research staff to total staff. 0\'erall, 
much of the work uses process variables as explanator\' var- 
iables in a c!a.ssic regression equation of indicators. This 
approach appears to do little to examine .sv-stematically the 
direct link between inputs and outconies, as the vast majorirs' 
of the applications of performance indicators, whether focused 
on student outcomes or research otitp.ut, de\'elop prox\- mea- 
sures of prcx-ess or conversion factors (Johnes and Taylor 
1990; Kells 1990 ). Some sticcess has been reported in devel- 
oping prcx-ess measures through periodic assessment of learn- 
ing environments (Sc-ottish Higher Education 1992). 

Based on the input prcxess output taxonomy, mo.st indi- 
cators are highly quantitative and lend themselves easily to 
.statistical manipulation, such as discriminate analysis of co.st 
measures (De. lager 1992; Johnes and Taylor 1990). A lack of 
satisfactor\- cost allocation procedures to deal with the mea- 
surement of either effectivcne.ss or efflcicnty remains, how- 
e\'er (Cave, Kogan, and Hanney 1990). Becatise of limitations 
on the data's stability, inconsistent operationalization, and 
weak relationships to articulated goals, indicators are viewed 
as part of the machiner\- of "management technolog> " (Segers, 
Wijncn, and Dochy 1990). 

A ho.st of decision makers and policy makers currentl>' con- 
sume data about performance indicators in Great Britain. .Since 
19H4. the government has been particularly intere.sted in 
improving the accountability of higher education in the eyes 
of taxpayers (Johnes and Taylor 1990). Tlie net effect of the 
emphasis was an increased demand for information from insti- 
tutions for use in allcx-ation of resources. Not only do govern 
ment decision makers use the data, however, as statistics 
about and performance indicators of university management 
have also found interest among consumers. Tlic British use 
these ciuantifiablc data in a table that compares in.stiuitions 
(Scottish Centrally F-iinded 1992). As a condition of devel- 
oping or using such national data bases, information require 
ments of different management le\-els mu.st be assessed and 
access negotiated (Si/.er 1992). 

While tho.se in British higher education were familiar w ith 
reporting .some type of performanc e claui, much ot the infor 
mation was used hi.storically more for adminislrative puqioses 
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than for management (Committee 19H5; Spec and Bormans 
1992). Thejarratt Committee's recommendation iaid the 
groundwork for a sliift in assumptions about governance of 
the academy and the resulting expansion of uses for perfor- 
mance data (Cave, Kogan. and Hanney 1990). Data u'ere used 
originally as management statistics for descriptive purposes 
and later to alkxate resources. In addition to the cla,ssic appli- 
cation of allcKating resources, .some have argued that the com- 
paratix'e assessment of institutions must not be a casual deci- 
sion (Bali and Wilkin.son 1992). When designed for u.se in 
alkx-ation, multiple measures are preferred ( Ca\'e, Hanney, 
and Kogan 1991). The relationshi|-) between performance indi- 
cators and peer rexiews should be complementary, with the 
former informing the latter (Sizer 1990b). 

Obser\'ers ha\-e begun to note that the .steady drift toward 
a bureaucratic model of higher education in Great Britain 
does little to benefit .students if pcdagog\- is ignored (Harnett 
1989; McElwee 1992). Whether intere.st in .students' learning 
will increase. e\-en if pedagogy- is not ignored, remains uncer- 
tain (Muffo 1993). 'file Committee of Inquiry- on Teaching 
Quality under the PoKtechnics and Colleges Funding Council 
could be one signal of a renewed intere.st in the learning 
environment (Sizer 1990h). Other recent eflbrts are under 
way to introduce extemal LLS.se.ssment of program tiuality 
through work .spon.sored by the Funding Council in England. 
.Scotland, and Wales. In addition, a .s\'.stem of quality audits 
is also under way through the Higher Education Quality Coun- 
cil (Yorke 199.^). and similar interest has been demon.strated 
by the Council for National Academic Awards (Yorke 1991a). 
The Quality Audit group, formerly known as the Academic 
Audit Unit, is another appr(.)ach to dex-eloping quality a.sse.ss- 
ment within the in.stitution. Reporting to the C\'CP. its role 
is to review methods for monitoring and promoting academic 
.standards among uni\-ersities ("^bung 1990). 

In 1992, macro performance indicators were developed 
for u.se with the former polytechnic .schools. Four .sets of per 
formance indicators were developed in the areas of effectix'c- 
ne.ss, resources, eftlcienty, and .source of funds (Kells 199.3; 
Polytechnics 1992). The Universities F'unding Council initiated 
a '.iirec-t link berween specialist panels and the use of indi 
cators. This approach represents a clear, complementary rehi 
tionship between cjuality ;isse.s.smcni through peer judgment 
;ind external indicators. Further dc\'elopment of performance 



indicators will be coordinated through the recently estab- 
lished Joint Performance Indicators Working Group, whose 
initial focus of interest is teaching quality', use of space, finan- 
cial health, research, and availability of data (Kells 1993). 

Great Britain has begun to redefine the role of higher edu- 
cation, and as it moves from an elitist to a mass s\-stem of 
higher education, governments are likely to demand more 
information to measure value. The academy now finds it.self 
competing for resources not only from government, but also 
from a marketplace filled with skeptical consumers. 



Canada 

Background. Covering nearly 3.9 million square miles with 
a population of more than 27.2 million by 1991. Canada in 
1991-92 ser\-ed a total of 554,021 students through 69 degree- 
granting institutions and 336,480 students through 203 pub- 
licly funded non-degree granting institutions (£!^rap« 1993). 
Enrollment in higher education increa.sed steadily from 1982 
through 1994 (A-sscKiation of Commonwealth 1993). Support 
for higher education is provided mainly through the provin- 
ces, nut becau.se the provinces varv- substantially in popula- 
tion, wealth, and, as a consequence, differential power, sup 
port for higher education has created an opportunity for the 
federal government to play an active role through the alio 
cation of re.sources. The poor economic climate in Canada 
has created lower estimates of revenue, increased nondiscrc 
ti{)nar\- expenditures, and reduced payme?nts to the provinces. 
Tighter budgets created a climate of concern about require 
ments for re.sources in Canadian educational institutions 
(As.S{K-iation of Universities 1993), and subsequent Rinding 
for "equalization" has created the concomitant call for ac- 
countability (A.ss{K-iation of Commonwealth 1993). Funding 
by the pro\ inces accounts for about 69 percent of university 
expenditures, while the federal government contributes 
another 13.5 percent and .student fees account for 10.2 
l^eiccnt. 

Environmental context. The subjei i of quality a.ssurance 
in general and the development of performance indicators 
in particular are best understood at the (irovincial and federal 
government level.s. At the [irovincial level, performance indi 
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cators f(X'us largely on issues of management; at the national 
level, the dominant project is the de\'elopment of descriptive 
statistics in a program kntwn a.s the Canadii effort, .supported 
by Statistics Canada/Council of Minister of Education (Cana 
dian 1992; Kells 1992b). 

Performance indicators have been developed in the context 
of the federal government's role in equalization, the region- 
alism of concentrated educational sen'ices, and the continuing 
is,sii.-s of bilinguali.sm and multiculturalism. Neither the uni 
\'ersities nor the colleges form an integrated "system" of 
higher education, and universities in particular enjoy con- 
siderable autonomy in Canada. 

Use of performance indicators. A ke\- emerging polic\- 
issue involves maintaining access while controlling educa 
tional co.sts. Attempts to balance these two forces will find 
financial aid policies at one end of the continuum and the 
government's policies on the alkx-ation of resources at the 
other. Some institutions plan to .sub.stantially increase tuition 
and fees as a way to reco\-er .shrinking support from provincial 
and federal gcnernments. According to the Ontario Task Force 
on Accountability; 

Al one L'lid of ihe spectrum is ivhat might he termed "strict" 
accoiiiitcthility, when, Jor example, utiirersities are held 
accountable hy goivniment Jor their expenditure of public 
funds by means of explicit rules and re^iulations. At the 
other end, there is what might more accurately he called 
"responsiivnes'!, " uiien, for example unirersities are said 
to be accountable to their local community for providing 
social and cultural leadership (As,S(x-iation of Universities 
1993. p. •4). 

In the early 19K0s, jucn-incial government.s began to inves 
tigate the role of universities in their regk)n.s. 'file Hove\- Com 
mi.ssion, for example, sought to understand the role of uni 
versitics in Ontario and their future plans. University 
responses laid tbe foundation for the development of per 
formance indicators that, in turn, were helpful in com[-)arati\e 
anal\■sc^s and fundraising (Snowdon 1993). 

Although federal and jiroxlncial go\ emments continue the 
dri\e to in.stall indicators, the media ser\v as an additional 
force. A/c/c/cYi//'.s- magazine ranked -K) uni\ersiiies in ;i report 
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that hixs furtlner politicized the atmosphere (1992). As in tine 
United States and western Europe, Canadian performance indi- 
cator systeins reflect the existence of an enormous vacuum 
in the marketplace of higher education. Government officials 
and consumers demand better information about educational 
.ser\ices, w hich can be provided by performance indicator 
systems ( Snowdon 1993)- 

As economic pressures mount and student enrollments con- 
tinue to increase, in.stitutional and .sy.stem performance indi- 
cators will shift from n focus on inputs to one on outcomes 
and their role in monitoring accountability and system per 
formance (AsstKiation of I'niversities 1993). At least five 
approaches to quality currently dominate the field— political 
economy, productivity, value added, producer consumer, con- 
tent—with some eclectic approaches as well (Nadeau 1992). 
Indicators of quality and excellence can be defined as "tho.se 
characteristics of ali components of colleges and universities 
for which there is some consensus as to their contributions 
to the succc.ssfi.ll attainment of specific in.stitutional mi.ssions 
and objective.s" (p. 2). 

The bulk of data collection nationally occurs in the federal 
go\'ernment's .Statistics Canada project. /Ml perfi)rmancc indi- 
cator programs rely on the Stati.stics Canada data ba.se for core 
information (Hawkins 1990). which includes data about .stu- 
dents, academic .staff, and finances. Data about .students in- 
clude .standard demographic characteri.stics and degrees 
granted. Faculty records include data on demographics, 
degrees held, compensation, and principal subjects taught. 
Financial data include standard total expenditure profiles by 
fiind and by type of expenditure and by function. XXIiile these 
three sets of files rely on the individual as the unit of obser- 
vation, each college data base aggregates the individual files 
into in.stitutional summaries. 

Provinces have develof.ed planning indicators to comple 
ment federal work. Among the provinces. Alberta's indicators 
emphasize in.stitutional efficiency and effectiveness; British 
Columbia u.scs traditional indicators to measure accounta 
bility, assess program quality, and gauge efilcienty: New- 
foundland's measures compare systems, manage po.stsec 
ondar\- education, and a.s.se.ss equity, efi"eclivene.ss, and ef 
ficienty, and Ontario is de\-eloping indicators in the context 
of enhanced planning (Hawkins 1990). Provincial indicat()rs 
will emphi'.si/.e public objectives, while federal ones will focus 
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primarilv on monitoring education using descriptive measures 
(Hawkins 1990). \X4iiie the need exists to incorporate mission- 
specific issues in the development of indicators, relatively lit- 
tle progress has been made. A partial explanation for the 
dependence on national management statistics is the sheer 
volume of data collected over the years; these national data 
sets may ha\e distracted academic polic>' makers from more 
serious in\ estigations of indicators. 

A major impediment to the use of performance indicators 
in Canada is the difficulty in building a defensible rationale 
for comparing institutions, although interinstitutional and 
interpro\'incial comparisons are beginning to be imple- 
mented. Much of the literature about Canadian projects 
underscores the need for taking context into account and 
developing a cla-ssification of institutions as a basis for con- 
■stnicting indicators. In addition, the development of inter- 
provincial comparisons remains suspect because integrity' of 
the data remains su,s|x>ct. Queen's University, however, devel- 
oped an approach to ranking and comparisc")ns with similar 
iastitutions, basing similarities on enrollment in undergrad- 
uate and graduate programs and the amount of sponsored 
research (Smith 1992). Hie success of the use of indicators 
at Queen's I'niversity might ha\'e been a function of senior 
management's quantitative orientation and the principal's 
interest in developing indicators (Snowdc^n 1993). Univer- 
sit\-widc indicators, however, might not necessarily provide 
sufllcient data to use in all(x.'ating resources (Nadeau 1992). 

The design of indicator .systems in Canada is similar to that 
in other de\'eloped nation.s, emphasizing primarily measures 
of either input or outcome. What little attention is paid to pro- 
cess factors is limited to traditional proxy ratio measures of 
con\-ersion, such as .student flow and length of studies. Equally 
significant, the measurement of process variables, defined 
b>' the learning environment, has not been .systematic. A 
recent highly critical report of teaching, for example, encour 
aged unilbrm .standards for evaluating teaching (Conference 
1991: Smith 1991). 

Attstrtlia 

Background. Australian higher education consists of a uni 
fied .sy.stem including some 40 in.stitutions. Unlike iastitutions 
in the I 'nited .States, all universities are .secular in origin and 
established through a( tion by the federal go\-ernmcnt (Asso- 



fiat ion iif Commonwealth 1993). 'ihe Australian Catholic Uni- 
versitN', although related to the church, is publicly funded and 
a member of the unified system. Australia alst : has a few pri- 
vate institutions and a handful of specialist institutions. 

Total enrollment in higher education increased substantially 
in recent > ears, rising from 336,702 in 1981 to 534,538 a 
decade later. In addition, distance learning grew to 56,922 
students in 1991. Empkiyment opportunities, however, do 
not show the .same pattern of growth, and unemployment 
coiintiwidc was estimated at 1 1.2 percent in late 1992. 

In 1992, the federal government allwated total funds of 
$A3.9 billicMi ($2.9 billion) to the universities. Funding occurs 
triennialK- on a rolling basis, with grants allcx-ated as a single 
blcx-k ( AsscK-iation of Commonwealth 1993). The common- 
wealth'.s support for research increa.sed to $A2.6 billion ($1.9 
billion) in 1991-92. 

Environmental context. Tlie growing concern about cjual- 
it\- in higher education in Europe appears to apply as well 
in Australia (Harman 1994; van Vught and Westerheijden 
1992). Beginning with the federal government's decision in 
198"' to develop a unified national .system, Australian higher 
education is changing dramatically (Goedegebuure and Meek 
1989). Tlie countr\-'s emerging sen.se of the need to broaden 
its economic base through a .stronger emphasis on interna- 
tional C()mpctiti\-ene.ss brought higher education into sharp 
fcx-Lis. Although the federal government did not have .specific 
powers over education, its financial support has set the direc- 
tion for higher education polity (Harman 1994). Intended 
to make higher education more responsive to the nation's 
economic de\-clopmcnt, the changes abolished the binarv' 
.SN-.stem. con.solidated in.stitutions. increased emphasis on ap- 
plied sciences, targeted re.scaich, altered staictures of gov- 
ernance, and jiromoted new ways to cut co.sts and develop 
altemati\-c re\ enuc .str-.-am.s. The primary- in.struments used 
to restructure higher education are financial incentives to 
reiluce the number of institutions through mergers. 

Au.stralian in.stitutions in the 19H0s depended on govern 
ment resoLirces to a great extent, which scn.-ed as an incentive 
to merge with other in.stitutions to meet minimum enrollment 
standards. M the same time, the federal government .sought 
to mo\'e from a highly regulated and centralized bureaLicratic 
.system to greater .self regulation, and a major change was the 
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introduaion of a unified national s\-.stem. The government's 
Higher Education: A Policy Statement { 1988) called for greater 
flexibility among institutions in programs and areas of re- 
search, as well as more control over resources and some 
reallocation of funds based on performance in priority areas. 
In sum, "institutional autonomy and flexibility [were] jux 
taposed with marketplace discipline" (Goedegebuure and 
Meek 1989, p. 221). Unlike in Great Britain, the Australian 
govemment's decision makers do not have a .strong commit- 
ment to the philosophy of a free market (leather 1990a). 

Use of performance indicators. Although considerable 
scholarship has addressed the relationship of performance 
to ciuality a.ssurance (Anwl 1992; Burke 1993a), the govern- 
ment's initiatives have been conventional. Performance indi- 
cators are classified into three t>pes: ( I) internal indicators 
that fcK'us on immediate po.stprogram performance (gradua 
tion rates), (2) operating indicators (cla.ss .size), and (3) exter 
nal indicators that fcx'us on students' .status after graduation 
(employment) (Commonwealth Tertian,- Education 1989). 

Like in Canada, performance indicators arc not limited to 
information about .stuclents. Comparisons of research among 
universities include such measures as the number of journal 
publications, grants, books published, editorship.s, and cita- 
tions (Grigg and Sheeban 1989). and the Performance Indi 
cators Research Group was commissioned to develop an 
initial li.st of indicators to measure efficiency and effectiveness 
(Lind.say 1993). These initial and subseciuent di.scussions 
t(K'u.sed on the development of information about perfor- 
mance as related to ciuality. 

An initial .set of 23 indicators was divided into three groups: 
( I ) institutional context, such as characteristics of the aca 
demic .staff, demand from students, their background, and 
cost data; (2) performance indicators, such as amount of 
teaching and learning, and research and professional ser\ ice; 
and ( 3 ) indicators of scx'ial equity, such ;ls gender ratios for 
.staff and .students ( l.inci.say 1993). A 1988 report of the Work 
ing Party recommends a series of indicators to measure in.sii 
tutional context and enhance the comparability of the findings 
across a range of institutions (Au.stralian 19KH). 

Most of the measures are conventional operalii )n;ili/.ations 
of input characteristics or outcomes. Some of the input v;ir 
iables can be conceptualized as measures of administrative 
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prcx'css. such as ratios calculated on the relationship of 
numbers of acceptances to actual enrollments (sometimes 
called a yield ratio in the United States). Other input variables 
can be viewed as intermediate outcomes. For example, 
second year retention and major sequence retention are com- 
ponents of student demand (Kells 1990). 

Traditional measures of outcome track undergraduate and 
postgraduate performance, such as proportion of completions 
and the number of completions within two years. Measures 
of "destination outcomes" fcx'us on employment status, post- 
graduate study, t)pe of Vvork, and salary. No data are collected 
to assess employers' perceptions of students' skills and 
competencies. 

Only one measure addresses teaching ciualit)'. which is a 
perceptual item based on a limited set of performance char- 
acteristics. As with mcxst ,s-)stems, the difTicult iasue of weight- 
ing perfonnance indicators remains unresolved. NX'liilethe 
initial report of the Working Party called for experts to de- 
\-elop weights, others ha\'e argued for public disclosure of 
a methodology that could include .statistical regression weight- 
ing (Hattie 1990). In addition, some concern has been 
expressed about the di.sciplinars' bias toward the .sciences that 
could cKTuras a result of re.search production indicators 
(Mo.scs 1990). 

Discipline reviews were introduced in the mid-1980s using 
senior academics, experts in x'arious di.sciplines. and major 
employers of graduates. 'Hie initial experience with these 
reviews has been mixed, and some \ iew tlie review prcx'ess 
as \ er\- costly and time consuming ( Harman 199-4; Higher 
Education Council 1992). A national academic audit mech 
anism is under de\elopment that is based on disciplinary- pan- 
els rather than external examiners. 

The (juality a.ssurance mechanism developed in Australia 
retlects a sensitivit\- to in.stitutional autonomy. A tiuality assu- 
rance plan developed by the Committee for Quality Assurance 
in Higher education (CQAHF.) began with a study of quality 
a.ssurance policies and pnictices. In 1994. ibe attention fo- 
cused on leaching and learning. The qualil>' re\ iew program 
is ,1 serious commitment to funding quality programs. In 199-4, 
for example, the program allocated $A"'6 million ($S5.9 mil 
lion ) to lho.se institutions that could "demon.strate effective 
(|uality a.ssurance polic ies and pnictices and excellent out 
comes" (O Neil 199-4). I'fforts in IWS should emphasize man 



agement as well as the integration of research and serN'ice. 
The CQAHE intends to assess outcomes at three levels: those 
asscKiatecl with institLitional missions, others related to the 
Higher Education CoLincil's goal statements, antt those reflect- 
ing the attributes of graduates and the impact of research 
(Undsay 1994). 

Tlie relationship between performance indicators and qual- 
ity remains underdev'eloped. When qualify' is conceptualized 
as the measurement of the "worth" of something, the ten 
denc7 is to limit the relationship to measures of outcome. 
TlTis narrow view disregards other complementars' avenues 
of connecting indicators to quality through reputation or con- 
tent (Lindsay 1992, 1993). The Higher Education Council in 
Australia has rai.sed the possihility of a national structure for 
quality assurance, and the National Board of Employment, 
Education, and Training published a paper in 1992 that lays 
the groundwork for an academic audit system that resembles 
work already underway' in Great Britain (Gordon 1992; 
Higher Education Council 1992). 

S(.)me interest has been expressed in revisiting indicators 
to fcx'us on "improving educational processes by attention 
to the many factors that influence student learning" (Lindsay 
1993, p- 34; see also Higher Educati<.)n Council 1992). The 
National Union of Students and the Council of Australian Post 
graduate Ass(x:iations endorsed the need for a broader per- 
spective of measures. 

Tine quality of the performance indicators developed to 
date has met with some criticism, including the underlying 
assumption of a single dimension to the measures (Hattie 
1990) and some argLiment that the existing indicators are not 
sufficiently substantive in their relationship to higher edu- 
cati(.)n\s g(.)als, jiiocesses, and outcomes (Lindsay 1993). Oth- 
ers have expressed concern about the use of performance 
indicators to introduce systematic formula funding (Au.stralian 
19H8; Hattie 1990). Subsequently, a higher education perfor- 
mance indicators research group was charged with deflning 
performance indicators operationally (C^ve, Hanney, and 
Kogan 1991 ), with performance indicat(.)rs recommended by 
the group to he applied .s-ystemwide ( Linke 1990 ). Another 
continuing problem is the lack o(a defensible a|iproach lo 
building valid comparison groups ( Linke 1990). 

[^e\'elopnients in Au.stralia mirror concerns expressed in 
other countries in apjilying a mechanistic \-iew of higher edu 



cation that produces measures of input, prcxess, and outcome. 
Concern has been expressed about the weakness of process 
measures, especially in the context of the complexity of 
higher education (Unke 1992). 

Few efforts are under way to addre.^.=; the difficult area of 
pr(K-ess measurement. One recent work refers to prcx-edural 
characteristics that might form baseline principles of good 
practice in some expression of formati\'e evaluation, and otTers 
a checklist of prtx-edures that constitute good practice in pro- 
cess (Hattie 1990). 'Hie notion of prcKess can be viewed from 
the perspective of managerial performance or characterfstics 
of the learning environment. VCIiether procedural differences 
explain \-ariations in student performance outcomes remains 
an unaddressed empirical question. Some have questioned 
whether prcKess measures should be used directly as per- 
formance indicators (Cullen 198"'). 

A major problem facing the development of performance 
indicators in Au.stralia is the a\-ailability of reliable data for 
comparative analysis (Doyle 1994; Kells 1993). A recently 
released major report has been developed on a set of equity 
indicators that augment the set of general performance indi 
cators (Murtin 1994). The report is the product of a project 
funded by the Department of Employment, Uducation, and 
Training to provide operational definitions for a wide range 
of performance indicators across functional areas within insti- 
tutions of higher education and .should provide considerable 
support for improved in.stitutional management information 
.systems. 

Tlie .swift movement to a unified .sy.stem. the number of 
mergers, and related polity changes have altered the land- 
scape of higher education in Au.stralia. It remains uncertain 
whether these change;; v^i 11 result in less diversity in programs 
and an increased reliance on quantitative indicators of .sys- 
tematic, in.stitutional, or departmental quality. The Working 
Party on Perf(irmance Indicators suggests that "to reduce funds 
as a penalty for poor performance would make the graduates 
cheaper but almo.st certainly worse" (Teather 1990b. p. 1 19). 
At this stage of performain-e indicators' de\-elopment in Aus 
tralia. the basic indicators will reflect stakelv ilders" perceptions 
of the u.ses and puipo.ses to be .ser\ ed by the .sy.stem. the 
financial dependency of in.stitutions. and the political ideology- 
of government decision makers. One can only speculate on 
w hether the goN crnmenl can tran.sform higher education into 
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;in economic and social engine for s>'stematic cliange. Recent 
funding patterns suggest that government decision makers 
will maintain a strong hand while allowing some participation 
in the definition of the journey. 

The NetherlaTtds 

Background. Dutch higher education has r^'o distinct seg- 
ments (Bormans et al. 1990). Universities prepare students 
for independent scientific careers in an academic orprofes 
sional setting, while HBO {hoger beroeps onderu-igs) institutes 
emphasize training in applied science for specific cxrupations. 
Currently. 20 universities, including se\-en theological in.sti- 
tutions, and 87 HBO institutes .ser\-e the Netherlands (Kells 
1992b). The Open Uni\'ersity was developed to provide uni- 
versity degrees, HBO programs, and other cour.ses. mainly 
through di.stance education. Go\'ernment funding of higher 
education accounts for approximately 90 percent of an insti 
tution's total budget (We.sterheijden 1990). Coie in.stitutional 
budgets are allcx-ated as a lump sum and used freely, while 
capital budgets are .strictly earmarked ( Frederiks. Wester- 
heijden, and Weusihof 1994 ). 

Environmental context. Since the carl\- 1980s, the Dutch 
government has developed a number of major polity pro- 
po.sals designed to re.structure its .strateg)' toward higher edu 
cation. A number of reasons are behind the new .strategy, 
including the ideological preferences of the political coalition 
in power since 1982 and the problematic budgetar>' situation 
facing the go\-ernment (Maa.sscn and van Vught 1988). Wiile 
these circumstances are similar to tho.se of other .six-ial demo- 
cratic countries like .Sweden and Denmark, the Dutch le- 
.sponse is unusual, because it directly links expanded in.sti- 
tutional autonomy with more stringent .standards of qualitv 
(Sander 1993)- One of the main .shifts in government polity 
toward higher etlucation is an increasing demand for greater 
rele\ance to .stx-iety ( Bormans et al. 1990). Tlie government 
wishes to encourage universities to suppK needed workers 
for .science and industr\- while generating new knowledge 
and technology- to benefit the econt)my. 

Tlie .Secrcary of Education and .Sciences issued a report 
on higher education, autonomy, and quality in 19Kt to 
emphasize ciuality in research and in.struction through 
increased accountability ( Drenth, Van Os. and Hem;Jert 1989 ). 




Tlie report called for a new approach to maiiaging higher edu 
cation for a number of reasons: 

1. Government regulations were leading to inflexible, cum- 
bersome procedures; 

2. histitutional responsibilities ^'ere too restricted; and 

3. Quality measurement and assurance were inadequate 
(D(X-hy, Segers, and Wijnen 1990bl. 

In 1988, the Ministr\' of Education and Science translated this 
broad government policy aito the Higher Education and 
Research Act, which contains the following components: 

1. Greater institutional autonomy in the design and iniple 
mientation of new curricula to encourage innovation and 
diversirv'; 

2. A system of quality assessment leading to public infor- 
mation on the ciuality of teaching and research at each 
institution to enhance competition; 

3. A .strengthened s\'stem of open access to all graduates of 
higher forms of scc'ondar\- education; 

4. Possible government response to an imbalance in the 
supply of student.s and the demand for courses offered 
and positions in the labor market b\' centrulK' reducing 
the number of students in a discipline; 

5. Maintenance of the government's role as principal funder 
of public higher education ( Bormans ct al. 1990). 

The SN'siem is administered by the Association of Cooperatiiig 
I 'ni\ersities ( the Vcmiifiing i wi Samemverkende Ncderlandsc 
I •mversik'itcii), a ciuasi public higher education ass(K-iation, 
as a forniatiw evaluation of study programs ever\' six years 
(Iongbloed;ind\X'cstcrheijden 1994). in 1993. the Higher 
Hducation and Research Act was codified into law, formally 
reducing go\ernment control over operational details while 
maintaining final responsibility for major issues of funding 
and ler.gth of program ( Goedcgebuure et al. 1993 ). 

Use of performance indicators. 'Ilic system of cjuality 
assessment was tested in 19KH in four disciplines (\Kestcr 
hcijden 1990). Tiie prtK-ess consi.sted of three major .stages: 
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1. Self o\aluati()n by faculties; 
^. Comparison among faculties by visiting review 
committees; 

3. Institutional response to the committee's findini;.s. 

In addition to selfstud\- reporus by the institutions, the Asso 
ciation of Cooperating L'niversities added a "fact book"" of 
centralized data that includes se\'eral performance indicators. 
Through the Higher Education and Research Plan, the go\-env 
ment intendetl to design performance indicators as opera- 
tional instruments for: 

1. Kvaluation to show the extent the goveninicnt"s goals ha\-e 
been ac hieved; 

2. Monitoring of relevant developments and trends; 

3. Dialogue of objecti\e information; 

■(. Funding (Ca\e, Hanney and Kogan 1991 ). 

The fact books v.-ere not well recei\ ed, because thc\- con 
tained incomplete data, di.screpancies with in.stitutional data, 
and ambiguous interpretations of data (We.sterheijden 1990), 
Consequently the asscK-iation e.stablished a .steering commit- 
tee to examine indicators and propose their future use in a 
new quality control .s\-.stem (Kells 1992b). Researchers, how 
e\-er. continue to tli.sagree on the use of perfonn:uice indi 
cators in (juality a.ssurance and debate their operational 
\'alidity. 

Others .support a .sy.stem of quality a.ssc,s.snient basetl on 
peer re\-iew rather than performance indicators (Woeijenstijn 
and Achernian 1990), suggesting the examination of the fun 
tlamental goals dri\'ing qualin- a.ssessment and reliance on 
a.sse.ssment tools consi.stent with tho.se objecti\-es. Wliile per 
formance intlicators might be perceived as helpful in meas- 
uring outcomes of governmental policies (with an orientation 
toward control), peer re\ iew pro\'ides the best ad\ ice for the 
enhancement of in.stifitions (an orien'ation toward improve 
ment) (Woeijenstijn and Acherman 1990). Adtlitional criticism 
of indicators includes '.ne lack of connection between indi 
cators and actual performance and the fear that indicators ulti 
mateK- become st;uitlards for performance. 

Others also call for a strengthen etl .system of peer re\-iew 
hut as.sert that increa.sed as.sessmcnt requires objectiw infor 
Illation ( Paartlekooper and .Spec 1990). Tliey do not consider 
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peer review and performance indicators incompatible ele- 
ments in assessment but beiie\-e tiiat judgments of qualiry- 
by experts can carr\- greater authorir\- if tiiey are based on 
knowledge of the facts. If assessments are to determine the 
future capacir\- of institutions in addition to the usual review 
of past performance, then it will be even more essential to 
include a sound empirical perspectiv e with the assessment 
made by academic peers. 

The most comprehensive debate surrounding development 
of the i ssessment plan has centered on the intrcxluction of 
performance indicators (Cave, Hanney, and Kogan 1991). 
Implementation has been difficult because of widespread 
skepticism about the validir\- and utilir\- of indicators and 
because of an absence of existing indicators for education 
in general rather than the predominance of performance indi- 
cators for research. A tentative list of 26 indicators was pub- 
lished in 198H with a ranking order of the uni\-ersities for 
each, but the list was critici/.ed because changes over time 
were reflected only in each institution's relative position com- 
pared to others (penalizing universities that performed well 
initially) and because i"" was unclear what some indicators 
were intended to measure. \C'hile the government might be 
pushing the use of performance indicators to achieve its 
higher education goals, the main polity thru.st remains the 
e.stablishment of .systematic self-assessment and external 
re\ icw (Ca\e, Hanney. and Kogan 1991 ). 

The publication ofthe.se initial indicators led to many sug 
ge.stions for further iniprcnement and de\elopn ent ot per- 
formance indicators: 

1. The purpo.ses of indicators should be explicit. 

2. Indicators based on clianges should not reflect badly on 
in.stitution^ with good early performance. 

3. Rimkings ba.sed on specific indicator; should be avoided. 
■i. r.se of indicators to dimini.sh neces.san di.scu.ssion v. ithin 

and ..mong in.stitutions should al.so be avoided (Bormans 
ct al. 1990). 

Further refinement of performance indi' ators will permit the 
government's withdrawal from management of prtK-e.sses, 
while gaining greater c < )r.[u il of product management ( Maa.sscn 
and van \u,ght 'lliis .shift in government planning and 

control will result in a more output oriented "mi.ssion fund 
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ing" mechanism for Dutch higher education. The government 
is willing to encourage the formative nature of quality assess- 
ment by not penalizing negative results from self evaluations 
or reports from pe jr groups (van Vught and Wcsterheijden 
1993). Only repeated deficiencies over multiple review cycles 
would result in decreased funding. 

Performance indicators currently are med \er)- little in set- 
ting Dutch higher education policy through quality- assess- 
ment Oongbloed and Westerheijden 1994). Indicators do play 
a critical role in funding, however, in determining the amount 
of core funds to award universities and HBOs, Since January' 
1993. a new s>'stem has been employed for funding core 
teaching ar.d research, with the following indicators directly 
determining a major portion of core funds appropriated to 
each institution: 

• Teaching indicators: Number of registered students; 
length of study; number of degrees av.arded 

• Research indicators: Number of dt Moral th.escs; number 
of certificates to design engineers. 

Thus, performance indicators play a very- minor role in shap 
ing discussion and polic-\- regarding quality assessment in the 
Nethedands, but a handful of indicators are critical in deter- 
mining core funding for higher education. 

Finlarui 

Background. Finland has 20 university-level institutions with 
approximately 100,000 students (Niiniluoto 1990), Organized 
national policy and a strong belief in national planning have 
characterized tlie Finnish system of higher education since 
the 1960s, Developments within higher education over the 
past 20 years include regional expansion, demcxTatization 
of university administration, mi.ssion-oriented lesearch pro 
mining scx-ial progress, and reform in curricula intended to 
narrow the gap between academic and vcx-ational education. 
NX-tiile each university has .some degree of autonomy in its 
internal affairs, the Mini.str\- of Hducation serves as the centnil 
unit of administration for higher education (Kells 1992b). 

Environmental context. Like many luiropean countne.s. 
the expansion of Finland's university- .system during the 1960s 
yvas partly motivated by- expanding democratization (Stolte 



Heiskanen 1992). This grcwing trend pul pressure on liigher 
education to broaden access to \^■ider segments of tlie pop- 
ulation and to provide a more diversified and qualified work 
Force, particularK- in scientific and teclmologicai fields. These 
new demands have resulted m a functional ovcrloatl, where 
universities attempt too many things without changing the 
structure of the institution and are simultaneously faced with 
increasing demands for efficiencs- and accountability. 

General planning for higher education is based on the Spe- 
cial Development Plan for Higher Education and the National 
Medium Term Economic Plan (Niiniluoto 1990). Tlie govern- 
ment approves the de\-elopment plan even,' four years, estab- 
lishing ov erall objectives for the development of higher 
education: 

1. Areas of acti\ it\- selected for emphasis; 

2. Specific development projects in re.search and teaching; 
and 

3. Changes in numbers of students. 

Each university also has a medium-term plan that is cooali- 
nated with a general economic plan for higher education and 
the national medium term plan (Hiifner and R;iu 198"^). The 
Higher Education De\-elopment Act of \9H~' guaranteed an 
annual increase of IS percent in funding for research, and 
po.stgraduate .studies during a five-year period— on the con 
dition that a university's performance be accountable and eval 
uated. Resource planning, efficiency, flexibility, leader'^'-'Sn, 
and acc( )untability ha\-e become key descriptors of ci 
higher education polity in Finland. 

Use of petformance indicators. The Higher Education 
Development Act of 19H"' directed all higher education in.sti 
tutions to apply a .system of performance e\-aluation resulting 
in sufficient and comparable data on outputs and co.sts of 
in.stitutional activities tCa\-e, llanney, and Kogan 1991 ). The 
Ministrv- of Education's working group on improving the 
methods of evaluating the performance of universities 
obser\ ed that a variety- of data related to uni\-ersiry produc 
tivitv had been collected and .stored by different authorities. 
Tlie group identified a number of problems with the exi.sting 
d;ita collection prcKX-.ss, including overlapping work, slow 
aviiilabiiity of information, difficulties in comparing ilat;i. and 
the lack of data use in planning ( Niiniluoto 1990). 
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In response to the recommendations of the working group, 
the Ministiy of Education c reated a data base to be used as 
a public clearinghouse for data about higher education. The 
information is reported by institution, with no departmental 
or individual data, and contains basic definitions. Available 
data from 1981 to the present include: 

1. Number of students 

2. Number of degrees conferred (masters, licentiate, 
dtKtorate) 

3. Staff by rank 

4. Costs 

5. Supplementan- training courses 

6. Space and facilities (Kells 1992b). 

Beginning in 198H, three data elements were added; sources 
of income other than budgeted funds, scientific publications, 
and research fellows working abroad. Tlie working group pro 
posed the creation (jf additional data bases at universities that 
would include certain information by department: number 
(jf teaching and research personnel, expenditures by depart 
ment. research financed b\' outside .sources, and .scientific 
activities. \Xliile the .scientific communirv- is critical and sus 
picious of the ii.se of performance indicators as a result of the 
lack ot prior di.scu.ssion concerning their use in evaluation 
(Niiniluoto 1990), the general .sen.se is th," indicators could 
prove to be usefLiI t(X)ls if developed and applied to increa.se 
the high quality of teaching and research (Kelis 1992a). 

Although the evaluation .system is .still being refined in Fin 
land, the principle of rewarding efficient in.stitutions and 
departments has already begun (Qtve. Hanney, and Kogan 
1991 ). Institutions receive additional funding as a reward for 
their prcKiuction of graduates who engage in research and 
postgraduate research training. .State officials are generally con 
tent to leave the educational .sy.steni alone and to effect 
change through marginal increments (.Stolte Hciskiinen 1992), 
As such, the development of performance indic ators in Fin 
land has f(K-u.sed on aggregate measures without serious con 
cern about institutional contexts and cK namics. Future 
attempts to develop indicators should fcK-us on pr(x-e.s,ses 
rather than inpiiLs and outputs. The governnieiit's top down 
ajiproach to developing performance indicators (llcilttii 19KK) 



results in no understantling of the complexities in educational 
prcK-esses or the many dimensions of outcomes in the pro- 
duction of knowledge. Tlie next few years will be critical for 
Finnish higher education as administrators attempt to find 
a balance between state and academic control. 

Sweden 

Background. Since reform in 19^'', Swedish higher educa 
tion has consisted of an integrated system th:" includes tra 
ditional university studies, former professional colleges, and 
a number of study programs previously offered by the second 
ar\- .school sy.steni (Bauer 1990). Currently, Sweden has 31 
institutions operated by the central government at 21 locations. 
(Furumart 1989). Tlie creation of a single comprehensive sys- 
tem of post,secondar\- education resulted from a need to 
reduce the differences in status bcwc^n postsecondar\' sec 
tors and to respond to requirements ^,1 rational planning 
(Kells 1992b). Tlie National Board of Universities and Col- 
leges scn'es as the central government authority for higher 
education and research. Approximately 184,000 .students 
attend Swedish higher education in.stitutions, an increase of 
25,000 from 1990 to 1993 (Sander 1994a). 

Environmental context. 'Hic National Board initiated a proj 
cct to foster self-study and self evaluation among departments 
and in.stitutions (Cave, Hanney, and Kogun 1991 ). Workshops 
that provided simulation exercises in trimming budgets high- 
lighted the difficulties in coping with conflicting goals and 
in establishing measures of quality while relating to harsh 
economic realities. 'Tliis move to incrca.sed decentralization 
has resulted in greater autonomy for universities and univer 
sity colleges, parlicularK- through increa.sed responsibility for 
planning, allcK-ation of resources, and clc\ clopment (Sizcr. 
Spec, and Bormans 1992). State budgeting systems are also 
being changed, resulting in more lump sum funding to 
institutions. 

This dcccnirali/.ation does not mean that the go\ernmcnt 
plans to relinquish all control of posl.sccondarA- education 
(Bauer IWO). Indeed, the intended change is ff ini manage 
ment h\ in.struction to managei-nent by objectives - - which 
reciuires more systeii.atic evaluation than has previousl\- been 
present in Swedish higher education. Responsibility lor 
f( )ll( .w lip, (juality control, and audits of aclivitv has al.so been 



given to individual institutions. Naturally. c\"iluation and 
methodologies have been high priorities for discussion, 
including the development and use t)f performance indicators. 

Swedish scholars are questioning the uses of institutional 
self evaluation (Furumart 19H9). Some contend that evalua 
tions .should not provide information to government author 
ities for their decision making, nor should it be a prcxess for 
institutional a.sse.ssment of efficiencA', quality, and performance 
to outside parties. They view self e\'aluation as an internal 
affair designed to provide a .sound basis for in.stitutional 
decisions. 

Use of performance indicators. Rather than building a 
completely new .system of evaluation for higlu-r education, 
authorities propo.sed that Sweden build on the better forms 
of evaluation already in existence ( Bauer 1990). A fundamen 
tal component of this .s\stem would be the National Board's 
annual report on higher education, which .synthesizes data 
.submitted by all universities and colleges on the u.se of 
resources and its results. The statistical data that ha\ c been 
regularly collected and jxiblished in the annual report 
include: 

1. Number of .student equivalents provided 

2. Number of .student equiv.-ilent.s u.sed 

3. NumbeT of annua! .student equivalents imn idcd 
-1. Number of annual s udent ec|uivalcnts u.sed 

Total credit production 
6. Total expenditures 
Total number of graduations ( Kclls 1992b). 

•Swedi.sh policy, however, does nv)t specilS' the kind of per 
formance information to be jirovidcd by the in.stitutions 
(Cave. Hanney, and Kogan 1991 ). Government legislation in 
bJSH called for the develoiimcnt of yardsticks to compare 
Swedi.sh higher education and research to other countries. 
The.se measure.s, however, do not rank in,stitutions aiul do 
not link quality control to iiroce.sses wiihin higher education. 

National elections in SeiMember 199 i marked the return 
to jiolitical control b\ .-.(R ial DenKu rats after a three year shift 
to a center right coalition and their move to ticcentrali/.e 
.Swedi.sh higher eclucaiion (Sander 1 Wia ). Ihe return ol 
scK iali.M leadership, however, will not eliminate the u.se of 



performance indicators and an emphasis on accountabilirs'. 
Rather, to allcKatc funds to institutions, the use of indicators 
like examination results and tlie numlicr of students graduat- 
ing will continue, while quantitative measures will be elim- 
inated from assessments of qualit\-. 

Denmark 

Background. Denmark, has tlve traditional universities, as 
well as 12 other institutions offering university-level instmc 
tion (Sander l99-ib). 'Flic higher education s\-.stem enrolls 
approximately 156.000 students, or about \=> percent of the 
college age population. I niversitN -level institutions can he 
public or pri\ate. with a Board of Governors overseeing the 
private institutions and a Rector appointed by the Ministry- 
of Education go\-erning the public in.stitutions ( Bache and 
Maassen 190.^). Government funding accounts for 100 percent 
of teaching expentiiturcs in higher education, with funding 
incrcasingh- mcn ing to a lump-sum alkK.-ation. 

Environmental context. Tlie University Admin i.stration Act 
of ITO gave significant influence to students in the go\-er- 
nance of universities. Critics called for .stronger management 
of the higher education sector, u hich resulted in Parliament's 
giving the Minister of Education pox\cr in ]9''6 to regulate 
•u-ce; ^ and fix .student numbers in broad program areas (Kells 
19921-) ). This initiative by the ministn,- has improved control 
of the budget anti helped balance higher education programs 
with the needs of the labor nrarkc Newrthele.ss. the current 
.system has a number of persistent weaknesses: 

1. Teaching and research are budget etl separately, but the 
MinistP,- does not know how the appropriati(.)ns are used. 
The .\tini.stP,- uses budget figures only, not accounting 
figures. 

1. No centralK- fixed teaching load exi.sts. and each collegiate 
body decides the amount of resources allcKated for 
teaching. 

3. OWcn the in.stitutions themselves are not aware of the con 
scquenies of their elaborate .systems to allocate teacl ing 
loads and resources. No accurate inform;ition exi.sts on 
how teachers .spend their working time. 
1. Between -lO antl W percent of professional resources are 
alkx-ated to research, but liltle is known about research 
output and the quality of research. 
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5. Tile Ministrv' i.s rcsiionsiblc for most of ilic builtlings, aiul 

it is not jiossiiilc to assess genfial costs aiui compart' them 
with similar information for other institutions. 

6. Tilt' Ministry- does not ha\ c easy access to information on 
mobility and age of personnel. 

^. No comparable Information exists on ,siud\- programs 
among the various in.siitutions (\bn l.instow 1990). 

In re.sponse to these concerns, ihe Ministr\- began to make 
the s-\-stem more tlexihle and resjionsix e to tiie needs of 
.society, while calling forgreai-r accoiintahilitv from univer- 
sities (Sizer 1992). In addit ion. the Mini.stn' created a general 
computer ha.sed infomiation .sy.stcm to .sene as a tool in the 
daily management of institutions and the o\ erall management 
of higher education. ^X^^ile the words "perrormance indii .i 
tors" were not u.sed at that time, the .\linistr\- wants to haw 
this type of infcjrmation (Kells 1993). 

State funding of higher education has been streamlined, 
with half of eac h institutional budget automatically awarded 
each year and the rest tied to the iiroduction of graduates 
(Sander 199-4b). This concern for the .small annual number 
of graduating .students was based on mo.si students' slow pace 
of completion and on the fret)ueni inteniiptions in their pn)- 
gram of .study. 

Use of performance indicators. In 1990. the Danish 
go\ emment released a plan. 'An Open Market for Higher Ldu 
cation." which emphasizes a super\ i.son,- role forgo\ernment 
and a decentralized, market driven .system for uni\ ersities and 
students ( Bache and ,Ma;LS.sen 1993 ). Tlie 1990 plan was cod 
ified in a )92 law reciuiring external evaluation of teaching 
t)ualin- in respon.se to decentralization ( Frederiks, Wester 
heijden. and Weusthof 199i ). This exchange of a.sse.ssment 
tor autonomy differs from the Dutch .system. howe\er. as Dan 
i.sh universities did not respond to go\ernment initiati\es. 
C;onsetiiiently, a gowmment funded indeiUMideni agenc\ \\as 
created to coordinate the quality as.se.s.snient .system (.Sander 
199.|b). The evaluation program is admini.siered by indepen 
dent educators and j tofcssionals in Denmark. .X major feature 
ot the a.s,se.ssment is the collection of data from .siudent.s, grad 
uales. and employers to ci.,, -niine ' i iistomer satislac!ion ' 
at a national le\el. hut no direct feedback isgi\en to in.sti 



tutions. To date, institutions liave strongly rejected attcmpis 
to incorporate pcrformunce indicators into evaluations (Sizer 
1992 ), and tensions will continue he^veen the government 
and institutions as long as the indicators are meant hoth to 
give internal insight and to provide information for externiil 
decisions (Sizcr, Spec, and Bormans 1992).Tlie Ministn,- has 
taken charge of the performance indicator project to ensure 
satisfactop.- solutions to the prohleni. The Danish experience 
illustrates the need lo collahorate and cooperate when de\el 
oping useful data that all parties can use (Kells 1993). 

A Comparison of International Indicator Systems 

Tahle S summarizes the use of performance indicators in (he 
seven countries discussed in this section. For each couniiA', 
nine comparatiw criteria are apiilied (see also Nedwek and 
Ncal I994h ). Tlic criteria arc not intended to assess the rel 
ati\ e merits of each approach hut to summarize trends hased 
on a\ailahle research literature. 

This .sample of initiatives mirrors a nuniher of themes dis 
cu.ssed earlier ( Nedwek and Ncal 1994b). First, indicators 
priniariK- are \-ariaiions of measures of input and outcomes. 
This mechanistic \-iew of data .selection could be dri\ en In- 
most initiatives' emphasis on accountability and hy the close 
connection between accountability and budgeting. Inputs 
and outcomes mo.st clo.sely resemble the emphasis on alio 
cations of mo.st appropriations prcx'e.sses. 

Second, a remarkable similarity exi.sts in the u.se of perfor 
mancc- indicators, regardless of the geographicu' setting. He 
spite differences in the size, .scale, and complexity of the Mate 
higher education system, the indicators and approaches to 
their u.se are ven.' similar in formal, f' .us. and di.ssemination. 
Fkit there are other similarities: 

1. Organizational control seems to he hierarchical during 
development of the model. Farly indications Imm Clreat 
Uritain suggest a mu\ emcnt to market e ontrol once ihe 
sNstc in matures. 

2. The primary use of performance indicators locuses on 
monitoring exi.sting systems. 

f). The models are built on indirect umnections lo siudeiii- 
learning, or an assumed leap of fiith thai ouiiomes uir, 
lie .uiiibuled I' i pr( k i-sses in the institution. 

' Wt- 
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TABLES 



NINE CRITERIA FOR PERFORMANCE INDICATORS 
APPLIED TO SEVEN COUNTRIES 



Critem 


Great Britain 


Canada 


Australia 


Netberiands 


Finland 


Sweden 


Demuailf 


1 bxusojComiOmMa 


Hierarchical wth 


Hierji'ihral 


Hietarchicil 


Chn and 


Hierarchical 


Hierarchic'al 


Clan 


archid. or marka emphasis 


market emphasis 






gwem mental 
hierarchical 








I De^eeujQxw.mndConmi 


Direii 


Direii 


Direct 


Indirea 


Direct 


Direa 


laissez 


Inidkmeni Direvt indirea 














(aire 


laissezEiire 
















3 Fwisoj PuTfiirmna hidaavrs- 


Effiiieiuy 


Ei\incne« 


Effeaiveness 


Efftciiveness 


Eificicncv 


Efficiency 


Effftiivc-ness 


iMituiional effean-eness. 




and effiiieniV 


and efficiency 










eificiena-. «wom 
















^ Svmesof Vanaioii III Qualit) 


Qimniui! 


Cummm 


Common 


&immiin 


Giinmon 


&immon 


Gimnion 


Emphasis ur speiial or common 
















ami 
















i Dm Sektmi. iktkds of 


Emphasizes input 


Inpub a-.d 


Input and 


liipuL'. and 


Outcomes 


Oiiiiomc^ 


Outcumes 


Gaiherm^Dmletvkol 




n'.ftnnie- 


• •aciime-' 


ouiccinic's 








Afpt^ioii Inpui, praess. 
















ouieomes 
















fi liiieiMAiidiemSsi Iniemal 


E\teiTa! wl\C\ make^ 


Internal 


Internal 


Exicmal 


E:<iemal 


External 


Extenul 


or external liec isionmakeR 




and external 


and external 


dftision 


polled 


dt\isi.)n 


policv 


and p>.i|ic\ makcB 




decision 
makers 


dec'isiur. 
makers 


tiuke^ 


niakers 


miers 


iMketv 


" Emphasis ojise Moniionng 


Monitoring and 


Monit'inng 


Mpniiimn? 


Miflitoniig 


Mimitoring 


Mcmiionng 


Monitonng 


mcisunng progress. forft"asing. 


jll'.uiini; 






and forcviiing 


and fiwastine 




diajjiosing. jIlMiing. manip 
















ulaiini; s\TnboLs 
















8 Imtuled Impdiioii iii(dmL< 


.teumed 


.\ssumed 


Indirfti 


Indirea 


.toumed 


.tounied 


.Assumed 


Leimng Direa. indirai. 
















assumed 
















Hekmishiptoliiilitulioiui! 


.VvumtO 


.teumed 


Indireci 


Direct 


Indirea 


Indirea 


Direct 



Mtsstoii Dim. indircM. • f 
assumed link between nicssiun 
and swm design 

'rills in-en'ievv of existing performance indicator systenvs 
helps clarify practices around the world. A study of five of the 
countries discussed in this section is the basis for the follow 
ing ten suggestions to facilitate funding agencies' develop 
ment and use of performance indicators in relationships 
between governments and institutions: 

/. // /.s hn/iiirtant for a f^oivmnicnt to sfwnci lintv clccirlv spec 
ijyin^ (i)td commiinkaliiig its oltjecliivs caul /nilicu's for 
tl.H'hifilur cciiicatiun sr^'stcm (hvjon'l clLscussiti^ with insti 
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tiiti(»is llx' clei vlopinciil and use of a s\'stL'in oj per/or 
iiieincc indicalors. 

2. C^oivrnmcnl should slimitlcUe dKci/ssiuus with imtitutiotis 
to cnaiuragc identification of institutional objectives in 
the context of gorerninent policies so as to identify areas 
(if agreement on criteria fur assessing achierenients 

,i It is important for a gorerninenl at an early stage to 
explain fully to institutions the uses . . . it intends to make 
of perfonnance indicators as part of the proces'^ of se- 
curing institutional acceptance of their use for assessing 
achiei enwnts. 

4. It is important to inivst time and resources on dereloping 
jointly with institutional representatii es a sound concep- 
tual basis for SI ihsequent deielopnwnt, including defining 
the nature, piiipose, and limitations of each indicator 

5. To build a reliable data base, it is important to start from 
the >ieeds for data within institutions . . . to secure an 
interest and active participation in the development and 
operation of the data base. 

(•>.... To lessen tensions that /could J arise in diverse and 
selective systems, it is a prerecptisite to make a distinction 
betn een the infornuition rec/uirements of different levels 
of nianagenieni (faculty, institution, and national ) and 
accept that not all managenuni statistics in the national 
data base should he available for use by national bodies 

7 // the funding model is based on equality of opportunity, 
funding, and comparable quality of institutions and 
courses. . . . the prinuiiy purposes of assistance to insti 
tutional nuinagers and the provision of relevant and re- 
liable data that /account/ for regional and special insti 
tutional factors /should he emphasized/. 

K /Institutions should he involved/ in the revieir of existing 
or the development of lunr resource allocation models. 

'). /It should he recognized/ that comparable teaching cpialit^' 
as<unince s\stenis in formed by agreed perfornuince indi 
cators are best developed by the institutions if their oivner 
.ship is to be secure(J or oirticrship is vested in a>i inde 
pendent body. 

10. It niii;ln lu' iiMtural iiml iK'ivssaiy l"( )r j^owrnmcnt to start 
iIk' proivs-s i)f ilcNX'lopiiij^ a national systL'ni ofpcrfor 
iiiain-c iiulii-aKifs htit alu^nvaii-l \cd\v it tn ihv institutions 
ol" liii;ln.T L'Liiii atioii it) lIi'Vl'!-)]! proposals on how tluy 
will iullill tlu' i-lcMiiands ( Si/.L-r mi. pp. 156 63). 



Summary 

International state liij^hcf eduaiticm .systems are diverse in 
size, scope, and mi.ssion, wliilc intcrnaticMial applications of 
performance indicators conversely present numerous simi- 
larities in the context, development, and use of such account 
ability measures. Increased fi.scal constraint.s. a cal! for rcle 
\'anc-y to the workplace, and a growing concern for return on 
the public's in\-estment have all shaped the development of 
performance indicator .sA'stems in the countries examineci in 
this .section. The final .section builds on this international ,sur 
\'ey to probe future implications for policy and practice. 



CHALLENGES FOR THE REST OF THE 1990s 



Limited resources, rising expectations, and a growing aware 
ness of the interconnectedness of educational SN'stems have 
implications for higher education and how its stakeholders 
will respond to heightened calls for public accountability. 
Tine lynic will dismiss or deny the moment, callii account- 
ability and cjiialit}- assurance yet one more fad foisted upon 
underpaid faculty by a burgeoning bureaucraty. The hard 
pressed, sincere policy maker will wonder whether perfor- 
mance SN'Stems impro\'e the quality of the learning environ- 
ment enough to justif\' their increasing costs. Other le.ss har- 
dened or le.ss conical observers, inside and outside the 
academy, might view the remaining halfofthe 1990s as a rare 
and narrow window of opportunity to reinforce and revitalize 
one of scK-iety's core functions— higher education. 

Tlie academy is beginning to understand that "getting back 
to b:isics" means far more than the three R's of clemcntars' 
and secondary education. It includes revitalizing a sense of 
academic integrity and, albeit reluctantly, providing more 
collective responsibility through a coherent, quality curricu 
him with an effective cadre of faculty. Expectations for au- 
thentic education, a revitalized faculty embracing new roles, 
and more learning for fewer government dollars have trans- 
formed the issues of the lO'^Cs and 1980s from access to 
concerns about and as.se.ssnient of productivity, c]uality, 
and accoimtability. 

We need to remind ourseK-cs that the academy, unlike pol 
icy makers and other .stakeholders in the conmiunity, has 
always .seen fii—by one means or another— to gauge how 
well it is doing. For nio.st of this centun,', higher educ-ation 
sought ways to determine qualit)' and to compare perfor 
mance among a .set of in.stitutioas. The current intercut in per 
formancc indicators, in a sen.se, represents another tool for 
measuring and monitoring quality. But it appears that the pur 
poses of performance indicators ha\ e changed o\ er the past 
half centur\'. Rankings based on reputation appear to have 
ser\-ed more .symbolic than informational expectations >)f 
management. Rankings scr\-ed as "brag sheets." pro\-iding 
only a Uxwe connection to performance (Dooris and Teeter 
199-4 ), and di.scu.ssions of reputation senvd as unolitaisive 
measures of influence within the academy. 

HerformanLC indicators were later useil to ini|-)rovc dei i 
sions about the allocation of resources. e.sixvialK through 
u.se of tools developed bv the National Cenlet for iligher Kdu 
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catitin Management Systems. More receiiily, indiattors ha\e 
been used lo design strategies for containing costs (Turk 
1992) and to increase efficiencA- in the use of resources (Riisii 
1992). Tfiese newer tools, iiowever, represent a significant 
departure from earlier mctlmdologies, especially techniciues 
to measure reput;uic)n, as performance indicators ha\e melded 
quality assurance with accountability to external stakeholders. 
Ha\'ing mo\-ed beyond s>-mboli.sni and indirect measures of 
influence, the ne»A-er tools have brought greater politicization, 
because the stakes are now much higher. 

In addition to the link to accountability, performance indi- 
cators appear to pro\-ide a more .serious appraisa' of j-serfor 
mance then the earlier methodologies. F.\en the more rigor 
ous approaches imdertaken by regir^nal or profe.ssional 
accreditation l^iodies have not enjoyed consistent support as 
the preferred nietht)clolog>- fc:)r achie\lng quality assurance. 
The "clubbiness"' of higher education (Ewell 1994a), a tra 
dition of highly abbreviated site visits, and the potential for 
politicizing .self .study icam.s in some regions of the I'nited 
Slates remain a problem. 

Short-term Future Trends 

Given this brief look at the recent pa.st and a .sample of appli 
cations on three continents, what can higher education adniin 
istralors expect in the future? Four recent trends will hcl|T 
shape the future of the academy. particularK' in the I'nltcd 
States, and di.scu.ssions about c|uality a.ssurance and account 
ability (Ewell 1994a). First, higher education will continue 
to be viewed as a key a.sset. Second, accountability in the K 
12 arena will remain more .specific and fcx'u.sed, while higher 
education will continue facing a crisis of confidence fmm pol 
icy makers. Third, the dominant focus will return to under 
graduate education, especialK' the general education com 
ponent of the curriculum and graduates' .skills. Fourth, higher 
education will remain driven by the market and higliK .sen 
siti\'e to community support. 

This o\-er\-iew of quality a.ssurance prt)granis antl perfor 
mance indicator initiatives in North .America, western Furopc, 
and Au.siralia suggests .sewra! other themes and treiuls What 
then lies ahead? Tlie remainder of this sec tion explores .some 
ofthe.se themes ami tin ; implications for the higher cdu 
cation community. 



rirsi. pcrtbrmaiuv indicators could achie\-e intlucncc 
beyond ihcir prepondcranl use as symbols to satisfy disccMi 
tented polity makers and consumers, becoming instead 
powerful tools that complement existing quality assurance 
practices. The explosion of in.dicators. opportunities for com 
paraiiN-e assessments, and increased suindardization ol key 
indicators and their definitions sugge.st greater use ot perfor- 
mance indicators in allocating lesources. 

Second, these new tools could be useful beyond meeting 
the information needs of governing boards and polity makers. 
It is reasonable to sugge.st that indicators could be u.seful tor 
consumers of higher education. Better information in the 
hands of iiarent.s, counselors, and .students remains in high 
demand. Similarly, industrial leaders as board members, pliil 
anthropic agencies with their cmphiisis on restricted giving, 
and bond underwriters see the potential for accessing belter 
(.lata on in.stitiitional performance. 

Third, as patterns of usage continue to evoKv, it appears 
that mandates for accountability are the primary fcKiis. while 
improvement to academic proces.ses remains .somewhat 
secondaiT, largelv because of difficulties with integrating such 
measures in the design of performance indicators. Total tiual 
ity management and continuous pnx'ess improvement held 
promise as potential proxies of a.sse,ssing academic process, 
but the\- ha\ e failed for the mo.st part to become part ol man- 
agement cultures in higher education. Applications are far 
more successful in cla.ssic business functional areas than in 
pedagogic settings (Oooris and Teeter 199^; Teeter and 

Ur/.ier 1993 ). , . 

The future seems to support coniinued reliance on admin 
i.strati\e rather than professional accountability (Edgerton 
199.3a. 199,3b). The use of "league tables" in F-urope and 
North America (Webster 1992). for example, is creating an 
atmosphere of superimposed, external .standards ot excel 
leiice. Because so many .sy.stems of performance indicators 
senv to monitor and control operations, it is more likely that 
the second half of the 1990s will bring continued emphasis 
on bureaucratic accountability than on professional account- 
abilitv. Current incentixes provided to faculty do not encour 
age collecti\-e responsibility for quality in the academy. The 
facullN- »77/use data from performance indicators but only 
when' more incentixes exi.st for doing so. The best potential 
Ibr professional accountability might be through rcMsiting 
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perfornuiiu-e standards within the acadt-nn-. Decisions about 
rank and tenure, for example, tiiat arc more sensitive to the 
Iieightcncd interest in exeeiient teadiin« of undergraduates 
eoiiid help restore more faith and tiiist in tlie aeadenn- for 
students anti other consumers. 

Fourth, given the fads that have marked higher education 
nianagement circles— management b\' objecti\-e. zero base 
budgeting, planning programming budgeting s-\stems. for 
example— it remains highly unlikely that performance indi- 
cators in in.stitutions will create enough broad-based support 
to sustain a reform movement. If performance indicators are 
mtegratcd with broadei requirements for information, how- 
ever, it is po.ssiblc that they can become more useful its man- 
agement tools (Borden and Banta 199-4 ). 

As organizations and educational .systems move toward a 
network approach to the functioning of organizations, man- 
agers will need to adjust accordingK-. A networked organi 
zation will demand that managers: 

1. Respond more rapidly to new situations, such as demo- 
graphic changes; 

2. Handle conflict and uncertainty where lines of authority 
and decision making are blurred; 

3. I.earn higher-order .skills in analysis and conceptualization 

4. I^arn how to work in problem focused teams; 

5. Use .systems of measurement that are sensitive to coop 
erative working enx'ironments; 

6. .Share resources and nurture diffuse accountability; 

7. Use information technolog\' to mo\'e information to deci 
sion makers more rapidly and within narrower planning 
CN'clcs; and 

8. Obtain the tcchnolog\' infrastructure to support emerging 
data management s\'stems ( Hockart and Short 1990 ). ' 

Performance indicator .systems can .ser\e as potentialK- use^il 
tools in the emerging cooperatiw work enx'ironment.s. 

Performance indicator .systems can helji create greater atten 
tion to context and mi.ssion, articulating -iiles for deci.sion 
making in building comparison groups, making chita more 
actionable, and dexeloping a .stronger conceptual and theo 
retical base. The i.s.sue will be whether managers can be de\'el- 
oped rapidly enough to take advantage of the information. 
\Xere indicator .sy.stems to be used as quick .solutions to com 



p!ex problems, however (for example, aciing as threshold 
standards for decisions about funding), poiia' makers and 
decision makers would quickly find themse!\ es in a nonpro 
ducti\'e, adversarial environment. 

Fifth, the tension bcwecn further centralization or renewed 
intere.st in decentralization will remain strong. Greater cen 
tralization, particularly at the state le\cl, accompanied 'in- 
growth in the use of performance indicator .sA-stems appears 
more likely to continue in the I'nited .Stale.s, but a reversal 
of that approach could .soon appear in Europe. Part 11 of the 
1992 amendments to the U.S. Higher Hducation Act encour- 
ages the federal government tt) set standards in areas tradi 
tionally re.sen-ed to in.stitution.s, for example, defining .siu 
dents' satisfuctor\- progress as taking ISO percent of the 
published time to complete a program. Schools mu.st also 
demonstrate ihat less than a third of their undergraduates 
withdraw during the regular academic year. In addition, slate 
regulatory language mu.st include ciuantifiable .standards for 
completion and graduation rates. Hcjually significant, accred 
itation officials are empowered to conduct unannounced site 
visits to all in.stitutions with vtK'ational programs 

Although the U..S. Department of Education retreated con 
siderably from earlier drafts of the regulations for Part i 1. the 
greater federalization of higher education in the I'nited States 
is likely to continue. Wliile .state po.stsecondaiy rex iew entities 
are mandated to develop specific indicators to monitor higher 
education institutions at the .state Icx'el, standards set b\- SPREs 
must be reviewed and appro\-ed by the federal goxernmenl. 
And other federal legislation reinforces the theme of central 
ization. For example, the .Student Right-to Kn(W and Campus 
Security Act requires reporting data about graduation rates 
and freciuemy of crimes' (xrurring. Polity makers' percptions 
that higher education has not adeciiiatek- reformed it.self inter 
nally and that supporting agencies like accrediting bodies are 
not up to the task have encouraged this shift in the locus o! 
control. 

Several European dex elopments. in contra.st. suggest 
renewed intere.st in quality a.ssurance models that inxolve 
members of the acadeniy. Quality audits b\- peers, .somew hat 
akin to vi.sits by accreditation teams in the I'nited .States, 
remain a x'iable way for an in.stitution to a.sscss pertorman(.-c. 
It is difficult to .speculate whether external performance indi 
cator .systems can be used to complement the work ol such 



auditing icanis. It might well be that different t\pcs of insti- 
tutions will ix' cncouragcti to dewiop institutional indicators 
or to select measures from existing national indicator systems 
that fit their distinct n.iission. llie demand for increased qual 
ity while offering increased autononn' in countries such as 
the Netlicrlands certainly jirovides attractix'e alternatixes. ^X''ith 
the exception of a few outcome measures associated with core 
funding decisions, jierformance indicators are little used by 
the Dutch. More important, the number of indicators is kejit 
rclati\el\- small, reflecting the gox'ernment's view that its in- 
\-olvement in the affairs of higher education should be kept 
to a minimum (jongblocd and V/e.sterheijden 199'i), 

.Sixtli. acce.ss to administratix'e [lerformance .systems through 
networks within and ainong coiitinents will accelerate in the 
la.si half of the 1990s, Developments in hardware and .software 
can nowsupjiort acce.ss to high volume transaction [-jrcK'e.ssing 
and the creation of mammoth frozen files. Hi.storically dis 
[larate data bases are bei-oming acce.ssiiile througli new Ian 
guages. 'Ilie.se rapid technological changes will increase the 
jiotential for ii.se of .systemic f)erformance indicators acro.ss 
a range of core in.stitutional fimctions and liK-ations. 

Seventh, awarene.ss is growing about the seamle.ssne.ss of 
all ediication--K- 12 tbix)iigh po,',t.sec,)ndar\-- and acro.ss po 
litical borders, but it is unclear whether future indicator .sys 
tems will be more intimately linked horizontally or vertically. 
Although discu.ssions are under way in elementary and sec 
ondaiy .schools, community colleges, and universities about 
educational problems in the United .States, resulting commit 
ments to collaboration in the areas of acce.ss, curriculum, and 
develojimcnt might not nece.ssarih' extend to integrated indi- 
cator .systems, A more iikel\- situation would inxoK'c efforts 
to deveioji national indicators with direct links to .schools and 
.systems that would be viewed as micromanagcmcnt among 
lower lex'cls of educational .systems. 

Performanie indicator .systems, howewr. might become 
more .sen.sitive to the interdependence of all Icx'els of edu 
cation. For example, measures of .students" performance in 
community colleges often include continuation rates to a four- 
year in.stitution as a "positive" outcome (American As.scK'iation 
of Community 199-4 ), I'or K 12, the Goals 2000 Act establishes 
voluntar\' national goals in the areas of. among others, .stu- 
dents' achievement, teachers' jireparation, adult literacy, and 
|iartnershi|is between .school and home. 



CDmpariSDiis of" performance could maw to an interna- 
tional level involving multinational comparisons similar to 
those de\-eloped by the OECD and, more recently, by the 
National Center for Education Statistics ( 1994a). A clear move 
nient has evolved toward developing core performance indi 
cators that can be applied across the globe. Such measures 
as student outcomes, retention, time to degree, graduation, 
employment patterns, and continued education can be used 
across nations. Indeed, the indicators used by the various 
countries examined in tliis volume are comparable, thus invit- 
ing comparison if problems concerning definition are solved. 
Such measures could hold special promise if such measures 
of outcome can be adjusted for characteristics, for example, 
.student.s" readiness (Astin 1993b). 



Impediments to Further Development 

Although performance indicatois potentially can impro\-e the 
quality of decision making, several (jbstacies to further refine 
ment remain, l-irst. the sheer number of indicators in some 
settings can scire more to confuse decision makers than to 
simplifN- the task of making ciioiccs. \>C'inona State I'niversity. 
for example, tracks 2S(j measures (\X'inona 199-4b). the I'ni 
\ ersity of .Miami more than 100 (Sapp 1993: Sapp and Temares 
1991. 1992 ). How to convert this plethora of data lo infor 
mation thai decision makers can use remains problematic. 

Second, issues of the validity and reliability of data gener 
ated from existing indicators mu.st be addressed .systematically, 
especially where performance indicators are part t)f allcK-ating 
resources. Recognition is growing that tjuality a.ssurance and 
accountabilitN' systems, such as those implied in most indi- 
cator models! cannot be designed to be applied independent 
of context. Institutional representati\-es and architects of per- 
formance systems are beginning to recognize the importance 
of mission and institutional context. From .selecting to oper 
ationalizing indicators, decision makers arc becoming more 
aware that context makes a difference. Tliis heightened sen 
sitivitN' has already created considerable discussion about core 
performance .standards needed for comparison across in.sti- 
tulions, ,s\-stems, states, or countries. The problems asscKiated 
with statistical or physical control of contextual variables illus 
trate the substantial methcKlological limitations that surround 
components of outcomes research of performance indicator 




initiatives (Pascarelia and Terenzini 1901 ). Outcome mcxsiircs, 
for exaniplf, remain a concern because of proiiiems of "con- 
textual validitN'" associated with any generic al-)iiic\- constaict 
(Mentkcwski and Rogers 1988). 

NX-liile context remains a key factor in shaping the design 
of performance indicators, it is important to distingiiisii fiscal 
measures from indicators of academic process and outcomes. 
'Die language of accounting and the mles of logic that ha\'e 
long .ser\'ed measurement of fi.scal accountability have nur 
tured a standard v.K-abulaiA' within the fiscal world. Calls for 
sensitix'ity to context are more appropriate for performance 
accountability. Mission-driven indicator in.stallations, such as 
SL'>A''s SN'stem and campus performance indicator,-,, can be 
designed to reflect contextual factors surrounding goals for 
access while using business and financial measiires to a,sse.ss 
funding (Fiurke 1993b). 

Tliird. as this review has shown, a leap of faith exi.sts be 
rwecn concerns over input or contextual characteristics and 
outcomes (Nedwek 199-+). This leap of faith represents the 
untested assumption that the educatior.al .system and its in.sti 
tution.s, programs, or culture explain variations in outcomes. 

Few signs exist that a consensus is emerging on the dimen 
sions and indicators of pr(K-ess variables or the role the\' 
should play in ciualir\' assurance ( I ;.S. Dept. of Education 
1992). Tills lack of agreement over priK-e.ss measures is more 
apparent at the system le\el than at the institutional le\el, 
where public interest in the learning environment could be 
more Rx-used. Equally important, at a lower le\'cl, analytic 
results from the use of indicators can yield more altematix'e-- 
for polity. Some research, for example, has reported that what 
happens in the classroom is asscK'iated with cognitix'e growth 
(National Center for Higher 1993b). Opportunities for active 
learning, freciucnt feedback on performance, and peer inter- 
action appear to influence the development of higher order 
thinking .skills (A.stin 1993b; Light 1990; McKeachie et al. 
198"'). Critics of professional and regional accrediting asso 
ciations in the L'nited .States note, howevei, that the current 
emphasis is on the act of assessment rather than on how the 
learning environment is improved (Nichols 1991a 1991b) 
Tlie irony is that while prcK'e.ss phenomen;; are related to out 
comes, the\' do reciuire considerable investment in re.sources 
for gathering data. In addition, the methodological or politic-al 
ccxsts iLSS(K'iated with gathering data lessen the likeliliood that 
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thcv will become a standard area of interest in performance 
indicator s\-stems. 

Process issues at the macro level have attracted lecent atten 
tion through the de\ elopnient of proxy administrative mea 
siires. Liunched in 1992, NAQ'BO's Benchmarking Project 
is one of the more ambitious attempts to apply multiple per 
formance indicators across 3H functions and procrsses in 
higher education (Kempner and Shafer 1993). Another inter- 
est^ing development is the biennial sur\'ey of strategic indi 
cators conducted by AGH in collaboration with Peterson's 
Guides (Taylor et ai. 1991). 

It is unclear whether process measures will be more widely 
found in performance indicator systems. Vie know that far 
more attention is eurrently given to outcomes and outputs. 
PrcK-ess measures might be more useful to decision makers 
ui the program level than to policy makers inside or outside 
an in.stitution, sy.stem. or nation. The perceived utility of pro 
cess measures could be more likely to increase when infor- 
mation .serves formative evaluation goals. Sorting out the u.ses 
of indicators by \ arioirs stakeholders could yield greater vis 
ibility of prcK-ess measures but little integration with mo.st 
existing u.ses of performance indicators. 

Fourth, faced with the temptation of complexity, added 
administrati\e costs, resi.stance from faculty, and a less for 
giving political culture, the probability increa.ses that by the 
turn of the centuiy mo.st indicator .s-^'stems will be viewed as 
too co.stly and ineffccti\-e as primar>- management tools. In 
the United States, for example, admini.strative and support 
co.sts alreacK- represent nearK- 30 percent of public education 
expenditures ( Haujitman 1990 ). 

Fifth, in the tradition of the individualistic frontier .settler, 
a hands off mentalit\ has long lingered as part of the academic 
culture in the United States. Despite the rituals of decisions 
about rank and tenure, we have held fa.st to the c-oncepts of 
the independent, individual .scholar and of academic freetk)m. 
Hie contribution to and responsibility for creativity in the 
academy have so far been largeK' accomplished through indi 
vidual effort. Performance indicator .s\stems do not fit the ciil 
ture of indi\ idual workers well. A primarv' challenge for the 
future is to redirect institutional attention, in palatable doses, 
to more collective re.sponsibility for re.stonng vitality and con 
fidence in the academ\-.Too .strong an emphasis on indicators, 
particulark b\- external agencies, howe\-er, could create an 



unintended baeklash among faculty, reinforcing individualism 
at the expen.se of .shared aecountabilin-. The resulting antag- 
oni.sm would likely harm the ver\' element that .such refornxs 
.seeic to improve -undergraduate education. 

'...e tension bet^veen individual accountability and collec 
tive responsibility is certainly no: limited to faculty Orga- 
nizational cultures, for example, can influence administra- 
tive i:)ehavior as well (Cameron and Ettington 1988; Peterson 
and Spencer 1993). Reengineering higher education includes 
moving from external controls to individual autonomy, from 
centralized .sy.srems to distributed networks, from hierarchical 
structures to horizontal networks, and from adult-child to 
adult adult professional relationships (Penrod and Do- 
Icnce 1992). 

lncenti\'es to nurture a sense of collecti\e responsibility 
in the culture cjf higher education have not been forthcoming 
in large doses. W:ether new wa\-s of viewing education- - 
as interconnected and mutually dependent components of 
a larger .sA'stem— will promote such incentives remains un- 
clear. Current methocfs of funding and managing education, 
especially the u.se of di.stinct re\-enue and expenditure streams, 
argue again.st collecti\-e re.sponsibility acrcxss educational 
sectors. 

Sixth, despite greater technological capacit\- to improve the 
efficiency of performance indicator .s\-.stems, the fcK-us of the 
indicators remains an i.ssue. Critics of indicator .systems sug- 
gest that current efforts fail to develop and apply a conceptual 
framework that is "based on research results and the interests 
of policy makers and educators" (Blank 1993. p. 67). Others 
ha\e argued that mcxst indicator sy.stems are too ambitious 
and unfcK-u.sed and lack \-alidity and reliabilit\'. Perhaps the 
best that can be .said is that the technolog\' to collect perfor- 
mance indicators exi.sts, but the evaluation of the results is 
yet Hawed. 

In sum. maintaining the balance between clemands for pub- 
lic accountability and internal qualit\- a.ssurance in the last 
half of the 1990s will not be ea.sy Growing pressures and a 
focus on centralization have created a tendeney toward greater 
inter\-ention by parties beyond the campus. 'ITie pace of such 
in\()lvemert in the early 1990s suggests yet more external 
influence, particularly in the United States. The nece.s.sar\- 
independence of the campus is certainly not threatened, but 
the .same le\'el of flexibility as in p;ist \-ears will be difficult 



10 sustain. A major objectK'e for campus and system admin 
istrators should he to respond to the new realities and place 
the institutional stamp and style on state or federal fiscal poi 
icies and education goals. 'Hie key is to strike a halance 
betNveen camiuis autc^nomy and government acccuintahilit>- 
for all activities. To that end. the experiment is worth 
continuing. 
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